Wednesday, 2017-02-15

ianwyeah, was good to learn a bit about that as i followed along.  previously i had some spring experience but on tomcat00:00
*** Swami__ has joined #openstack-infra00:00
*** jcoufal has joined #openstack-infra00:01
*** vaidy has joined #openstack-infra00:01
*** thorst_ has joined #openstack-infra00:02
*** jamielennox_ has joined #openstack-infra00:02
*** thorst_ has quit IRC00:02
ianwfungi: i mentioned before, but i think it went into a black hole - i wonder if https://docs.openstack.org/user-guide/search.html?q=neutron was broken before the https switch?00:03
ianwjust clicking i noticed the release notes have mixed content -> https://review.openstack.org/#/c/433962/ but didn't notice anything else after quick poke00:03
*** sflanigan has quit IRC00:03
*** Swami has quit IRC00:04
*** Swami_ has quit IRC00:04
*** pahuang has quit IRC00:04
*** jak-jak has joined #openstack-infra00:04
*** Swami has joined #openstack-infra00:04
*** jamielennox_ is now known as jamielennox00:06
*** hyakuhei has joined #openstack-infra00:06
fungiare you getting the same "please enable javascript" error i am there then?00:07
ianwyep00:07
fungimy guess is it's something to do with the gcse.src conditional there00:07
ianwLoad denied by X-Frame-Options: https://cse.google.com/cse_v2/ads?q=neutron&... blah00:07
fungioh, i see. so it's using https correctly just not being allowed00:08
*** annegentle has quit IRC00:09
ianwalso a 404 for https://docs.openstack.org/user-guide/common/js/doc-characters.json00:09
fungihttps://github.com/google/google-api-javascript-client/issues/209 maybe?00:10
*** dingyichen has joined #openstack-infra00:10
*** rm_work| is now known as rm_work00:11
*** rm_work has joined #openstack-infra00:11
ianwhmm, interesting, in chrome, it pops up a search box00:12
ianwnot in firefox00:13
*** jascott1_ has joined #openstack-infra00:13
*** Goneri has quit IRC00:13
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: TEST: Don't override worker counts  https://review.openstack.org/43399200:14
*** andreww has quit IRC00:16
*** EricGonc_ has quit IRC00:16
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: Test with bonded public interface  https://review.openstack.org/38556200:17
fungiianw: should probably bring this to AJaeger's attention once he wakes up, and feel free to emergency revert 432334 for now if needed, we can always revert the revert later once this is fixed00:18
*** ijw has joined #openstack-infra00:18
openstackgerritMerged openstack-infra/tripleo-ci master: Revert "Revert "toci_gate_test: allow to run upgrade jobs for scenarios""  https://review.openstack.org/43371500:19
ianwfungi: yeah, i'm not 100% sure it's related?  and it mostly works with chrome (although you still get that odd no js error).  so yeah, just a bug to be handled in due course i'd say00:19
*** rbuzatu has quit IRC00:20
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: Drop ceph workaround for mitaka  https://review.openstack.org/42575400:20
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: Remove rdo-qemu-ev workaround  https://review.openstack.org/42575300:20
*** sflanigan has joined #openstack-infra00:21
fungias long as our infra liaison for the docs team is aware, i'm cool00:21
fungiwe can treat it as severely as they need00:21
*** pahuang has joined #openstack-infra00:21
*** mtanino has quit IRC00:23
*** jcoufal has quit IRC00:24
*** xarses_ has joined #openstack-infra00:25
*** rbuzatu has joined #openstack-infra00:25
*** john-davidge has joined #openstack-infra00:28
*** Julien-zte has joined #openstack-infra00:29
*** rbuzatu has quit IRC00:30
*** xarses_ has quit IRC00:31
*** john-davidge has quit IRC00:33
*** xarses_ has joined #openstack-infra00:33
*** xarses_ has quit IRC00:33
*** xarses_ has joined #openstack-infra00:34
*** dimtruck is now known as zz_dimtruck00:36
*** hongbin has quit IRC00:37
*** EricGonczer_ has joined #openstack-infra00:38
*** EricGonczer_ has quit IRC00:38
*** xarses_ has quit IRC00:42
*** rbuzatu has joined #openstack-infra00:43
*** Guest3904 is now known as medberry00:43
*** medberry has quit IRC00:43
*** medberry has joined #openstack-infra00:43
*** medberry is now known as med_00:43
*** annegentle has joined #openstack-infra00:44
*** denisra has quit IRC00:44
ianwclarkb: if you got a minute, wouldn't mind some thoughts on capturing the coverage output for openstack-health as suggested by fungi.00:47
ianwi'm thinking we have to shoe-horn the stats into the subunit output, maybe00:47
*** sree has joined #openstack-infra00:47
*** tlian has quit IRC00:50
*** sree has quit IRC00:52
clarkbianw: I think that is what would be required if we don't want to change openstack-health at all00:52
ianwyeah, first step i'm just reading up on subunit streams etc to see if we can extract the overall coverage number and insert it00:54
clarkbianw: we could potentially modify the coverage command that recombines all the sub coverage to emit subunit00:54
clarkbor even just have our own little wrapper around coverage that did it00:54
ianwyeah, i think i'm coming to the latter, to start with anyway00:54
ianwi guess putting coverage in post and tracking there does make sense to get that "moving average" type overview00:56
*** tlian has joined #openstack-infra00:59
*** gongysh has joined #openstack-infra00:59
*** tphummel has quit IRC01:04
*** xarses_ has joined #openstack-infra01:09
*** thorst_ has joined #openstack-infra01:10
*** andreww has joined #openstack-infra01:10
*** thorst_ has quit IRC01:10
*** andreww has quit IRC01:10
fungithat's sort of what i was thinking as well. if we can turn coverage values into subunit somehow...01:10
*** andreww has joined #openstack-infra01:10
*** thiagolib has joined #openstack-infra01:11
*** smarcet has joined #openstack-infra01:12
*** xarses_ has quit IRC01:13
*** gongysh has quit IRC01:14
*** shu-mutou-biz has joined #openstack-infra01:14
*** wolverineav has quit IRC01:16
*** Apoorva has quit IRC01:17
*** zz_dimtruck is now known as dimtruck01:18
*** smarcet has quit IRC01:19
*** liujiong has joined #openstack-infra01:23
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs  https://review.openstack.org/35921501:24
*** yanyanhu has joined #openstack-infra01:24
*** pvaneck has quit IRC01:26
*** MasterOfBugs has quit IRC01:26
*** sdague has joined #openstack-infra01:31
*** wolverineav has joined #openstack-infra01:32
*** wolverineav has quit IRC01:37
*** sdague has quit IRC01:40
*** annegentle has quit IRC01:43
bkerofungi: https://www.indiegogo.com/projects/gpd-pocket-7-0-umpc-laptop-ubuntu-or-win-10-os-laptop--2/x/5727346#/01:43
*** thorst_ has joined #openstack-infra01:44
*** andreww has quit IRC01:45
*** sree has joined #openstack-infra01:46
*** rbuzatu has quit IRC01:47
*** rlandy has quit IRC01:47
*** rbuzatu has joined #openstack-infra01:48
*** erikmwilson has joined #openstack-infra01:49
*** sree has quit IRC01:51
*** shu-mutou-biz has quit IRC01:51
*** liujiong has quit IRC01:51
*** liujiong has joined #openstack-infra01:51
*** rbuzatu has quit IRC01:52
*** xarses_ has joined #openstack-infra01:56
*** mdrabe has joined #openstack-infra01:57
*** thorst_ has quit IRC02:03
*** gongysh has joined #openstack-infra02:05
*** baoli has joined #openstack-infra02:08
*** rbuzatu has joined #openstack-infra02:08
*** jascott1_ has quit IRC02:11
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup  https://review.openstack.org/43373602:11
*** jascott1_ has joined #openstack-infra02:12
mriedemhttps://review.openstack.org/#/c/433685/ has been running in tripleo-experimental for nearly 11.5 hours02:12
mriedemseems something is off the rails there02:12
*** zara_the_lemur__ has quit IRC02:12
*** rbuzatu has quit IRC02:13
*** wolverineav has joined #openstack-infra02:14
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job  https://review.openstack.org/43164902:15
*** jascott1_ has quit IRC02:16
*** dimtruck is now known as zz_dimtruck02:16
openstackgerritBrian Haley proposed openstack-infra/devstack-gate master: Support an IPv6 underlay network  https://review.openstack.org/34304102:18
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job  https://review.openstack.org/43164902:18
*** sflanigan has quit IRC02:20
*** thorst_ has joined #openstack-infra02:24
*** thorst_ has quit IRC02:24
*** thorst_ has joined #openstack-infra02:25
openstackgerritGhanshyam Mann proposed openstack-infra/tripleo-ci master: Fix to use correct config options for ssh user  https://review.openstack.org/43402602:26
*** thorst_ has quit IRC02:29
*** thorst_ has joined #openstack-infra02:31
*** sflanigan has joined #openstack-infra02:31
*** wolverineav has quit IRC02:34
*** xarses_ has quit IRC02:34
*** zz_dimtruck is now known as dimtruck02:34
*** thorst_ has quit IRC02:36
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: tripleo/scenario004: add manila files  https://review.openstack.org/43403102:40
*** Sukhdev has quit IRC02:40
*** gouthamr has quit IRC02:45
*** tqtran has quit IRC02:45
*** gouthamr has joined #openstack-infra02:46
*** rbuzatu has joined #openstack-infra02:49
*** baoli has quit IRC02:53
*** rossella_s has quit IRC02:53
*** ramishra has joined #openstack-infra02:53
*** baoli has joined #openstack-infra02:53
*** rossella_s has joined #openstack-infra02:53
*** rbuzatu has quit IRC02:54
*** yuanying has quit IRC02:55
*** john-davidge has joined #openstack-infra02:55
*** john-davidge has quit IRC02:59
*** yamahata has quit IRC03:00
*** markvoelker_ has joined #openstack-infra03:02
*** amotoki has joined #openstack-infra03:02
*** rkukura_ has joined #openstack-infra03:03
*** baoli has quit IRC03:03
*** masayukig_ has joined #openstack-infra03:04
*** ewindisch_ has joined #openstack-infra03:04
*** dhinesh has quit IRC03:04
*** electrical_ has joined #openstack-infra03:04
*** robcresswell_ has joined #openstack-infra03:04
*** boris-42_ has joined #openstack-infra03:04
*** fmccrthy_ has joined #openstack-infra03:05
*** nikhil_ has joined #openstack-infra03:05
*** nikhil_ is now known as Guest1620003:05
*** ansiwen_ has joined #openstack-infra03:05
*** vdrok_ has joined #openstack-infra03:05
*** mdbooth_ has joined #openstack-infra03:05
*** szaher_ has joined #openstack-infra03:05
*** serverascode_ has joined #openstack-infra03:06
*** jbernard_ has joined #openstack-infra03:06
*** jamespag` has joined #openstack-infra03:06
*** tonyb_ has joined #openstack-infra03:06
*** mnencia_ has joined #openstack-infra03:06
*** slagle_ has joined #openstack-infra03:06
*** dmellado_ has joined #openstack-infra03:06
*** jamespag` has quit IRC03:06
*** jamespag` has joined #openstack-infra03:06
*** bradm_ has joined #openstack-infra03:06
*** njohnston_ has joined #openstack-infra03:06
*** _d34dh0r53_ has joined #openstack-infra03:06
*** Zara_ has joined #openstack-infra03:06
*** adam_g_ has joined #openstack-infra03:06
*** reed_ has joined #openstack-infra03:06
*** clayg_ has joined #openstack-infra03:06
*** admcleod_ has joined #openstack-infra03:06
*** nibz has joined #openstack-infra03:06
*** coreycb` has joined #openstack-infra03:06
*** Gorian- has joined #openstack-infra03:06
*** rosmaita_ has joined #openstack-infra03:07
*** cinerama` has joined #openstack-infra03:07
*** ramishra_ has joined #openstack-infra03:07
*** arif-ali_ has joined #openstack-infra03:07
*** mugsie|a1t has joined #openstack-infra03:07
*** phealy has quit IRC03:07
*** docaedo_ has joined #openstack-infra03:07
*** Hunner_ has joined #openstack-infra03:07
*** Hunner_ has quit IRC03:07
*** Hunner_ has joined #openstack-infra03:07
*** phealy has joined #openstack-infra03:07
*** njohnston has quit IRC03:07
*** nikhil has quit IRC03:07
*** vdrok has quit IRC03:07
*** mnencia has quit IRC03:07
*** yonglihe has quit IRC03:07
*** fdegir has quit IRC03:07
*** szaher has quit IRC03:07
*** dmellado has quit IRC03:07
*** arif-ali has quit IRC03:07
*** serverascode has quit IRC03:07
*** jbernard has quit IRC03:07
*** ewindisch has quit IRC03:07
*** fmccrthy has quit IRC03:07
*** electrical has quit IRC03:07
*** admcleod has quit IRC03:07
*** markvoelker has quit IRC03:07
*** boris-42 has quit IRC03:07
*** jamespage has quit IRC03:07
*** calebb has quit IRC03:07
*** smcginnis has quit IRC03:07
*** Zara has quit IRC03:07
*** slagle has quit IRC03:07
*** verdurin has quit IRC03:07
*** reed has quit IRC03:07
*** ramishra has quit IRC03:07
*** knangia has quit IRC03:07
*** tonyb has quit IRC03:07
*** rkukura has quit IRC03:07
*** adam_g has quit IRC03:07
*** Cibo has quit IRC03:07
*** apetrich has quit IRC03:07
*** rosmaita has quit IRC03:07
*** ansiwen has quit IRC03:07
*** bswartz has quit IRC03:07
*** coreycb has quit IRC03:07
*** robcresswell has quit IRC03:07
*** cinerama has quit IRC03:07
*** clayg has quit IRC03:07
*** nibalizer has quit IRC03:07
*** docaedo has quit IRC03:07
*** bradm has quit IRC03:07
*** masayukig has quit IRC03:07
*** gcb has quit IRC03:07
*** mdbooth has quit IRC03:07
*** Hunner has quit IRC03:07
*** lbragstad has quit IRC03:07
*** Gorian has quit IRC03:07
*** shadower has quit IRC03:07
*** d34dh0r53 has quit IRC03:07
*** pcrews has quit IRC03:07
*** mugsie|alt has quit IRC03:07
*** mnencia_ is now known as mnencia03:07
*** rkukura_ is now known as rkukura03:07
*** adam_g_ is now known as adam_g03:07
*** arif-ali_ is now known as arif-ali03:07
*** smcginnis_ has joined #openstack-infra03:07
*** smcginnis_ is now known as smcginnis03:07
*** gcb has joined #openstack-infra03:08
*** reed_ is now known as reed03:08
*** ijw has quit IRC03:08
*** apetrich has joined #openstack-infra03:08
*** lbragstad has joined #openstack-infra03:08
*** verdurin has joined #openstack-infra03:08
*** ewindisch_ is now known as ewindisch03:08
*** fmccrthy_ is now known as fmccrthy03:08
*** calebb has joined #openstack-infra03:08
*** pcrews_ has joined #openstack-infra03:08
*** Cibo has joined #openstack-infra03:08
*** vdrok_ is now known as vdrok03:09
*** robcresswell_ is now known as robcresswell03:09
*** yonglihe has joined #openstack-infra03:09
*** electrical_ is now known as electrical03:09
*** Guest16200 is now known as nikhil03:09
*** thorst_ has joined #openstack-infra03:10
*** rbuzatu has joined #openstack-infra03:11
*** fdegir has joined #openstack-infra03:11
*** thorst_ has quit IRC03:11
*** gildub has joined #openstack-infra03:13
*** jascott1_ has joined #openstack-infra03:13
*** dchen has joined #openstack-infra03:15
*** isq has quit IRC03:15
*** rbuzatu has quit IRC03:16
*** jascott1_ has quit IRC03:18
*** rosmaita_ is now known as rosmaita03:18
*** serverascode_ is now known as serverascode03:18
*** amotoki has quit IRC03:19
*** jaosorior has joined #openstack-infra03:19
*** tonyb_ is now known as tonyb03:21
*** masayukig_ is now known as masayukig03:21
*** ihrachys has joined #openstack-infra03:23
*** isq has joined #openstack-infra03:29
*** wolverineav has joined #openstack-infra03:31
*** wolverineav has quit IRC03:35
*** sflanigan has quit IRC03:35
*** thorst_ has joined #openstack-infra03:37
*** thorst_ has quit IRC03:37
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup  https://review.openstack.org/43373603:41
*** Jeffrey4l_ has quit IRC03:43
*** Jeffrey4l_ has joined #openstack-infra03:43
*** wolverineav has joined #openstack-infra03:43
*** gildub has quit IRC03:44
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job  https://review.openstack.org/42748103:45
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin)  https://review.openstack.org/42748203:45
*** ihrachys has quit IRC03:45
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup  https://review.openstack.org/43373603:45
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job  https://review.openstack.org/43164903:46
*** wolverineav has quit IRC03:47
*** rbuzatu has joined #openstack-infra03:48
*** sflanigan has joined #openstack-infra03:48
*** armax has quit IRC03:52
*** rbuzatu has quit IRC03:52
*** cshastri has joined #openstack-infra03:53
*** ihrachys has joined #openstack-infra03:54
*** sree has joined #openstack-infra03:55
*** docaedo_ is now known as docaedo03:59
*** yonglihe has quit IRC03:59
*** ihrachys has quit IRC04:01
*** gouthamr has quit IRC04:01
*** nmathew has joined #openstack-infra04:02
*** nicolasbock has quit IRC04:06
*** yuanying has joined #openstack-infra04:08
*** rbuzatu has joined #openstack-infra04:09
*** YorikSar has quit IRC04:11
*** thiagolib has quit IRC04:11
*** YorikSar has joined #openstack-infra04:12
*** rbuzatu has quit IRC04:13
*** dhinesh has joined #openstack-infra04:14
*** amotoki has joined #openstack-infra04:22
*** sbezverk has left #openstack-infra04:30
*** rbuzatu has joined #openstack-infra04:30
*** YorikSar has quit IRC04:30
*** yamahata has joined #openstack-infra04:31
*** YorikSar has joined #openstack-infra04:32
*** dimtruck is now known as zz_dimtruck04:32
*** links has joined #openstack-infra04:34
*** rbuzatu has quit IRC04:34
*** tphummel has joined #openstack-infra04:34
*** thorst_ has joined #openstack-infra04:38
*** jascott1_ has joined #openstack-infra04:43
*** thorst_ has quit IRC04:43
*** jascott1_ has quit IRC04:48
*** dhinesh has quit IRC04:50
*** shu-mutou is now known as shu-mutou-AWAY04:56
*** masayukig is now known as masayukig_04:58
*** Sukhdev has joined #openstack-infra04:58
*** wolverineav has joined #openstack-infra05:00
*** wolverineav has quit IRC05:05
*** adriant has quit IRC05:07
*** rbuzatu has joined #openstack-infra05:11
*** Sukhdev has quit IRC05:12
*** mdrabe has quit IRC05:13
*** tphummel has quit IRC05:13
*** rbuzatu has quit IRC05:16
*** wolverineav has joined #openstack-infra05:16
*** knangia has joined #openstack-infra05:18
openstackgerritIan Wienand proposed openstack-dev/pbr master: Remove testr entry point  https://review.openstack.org/42237905:22
*** john-davidge has joined #openstack-infra05:22
*** nadya has joined #openstack-infra05:23
*** zz_dimtruck is now known as dimtruck05:23
*** wolverineav has quit IRC05:24
openstackgerritIan Wienand proposed openstack-dev/pbr master: Also generate XML coverage report  https://review.openstack.org/43407105:24
*** nadya has quit IRC05:24
ianwlifeless: ^^ i think that changelog now better describes what the heck is going on with testr/test and PBR05:25
*** amotoki has quit IRC05:26
*** wolverineav has joined #openstack-infra05:26
*** john-davidge has quit IRC05:27
*** faizy has joined #openstack-infra05:28
*** wolverineav has quit IRC05:30
*** dimtruck is now known as zz_dimtruck05:34
jianghuawhi, is the zookeeper service mandatory for nodepoo-0.3.1?05:35
*** amotoki has joined #openstack-infra05:36
*** sandanar has joined #openstack-infra05:36
*** sflanigan has quit IRC05:40
*** bradm_ is now known as bradm05:40
*** dingyichen has quit IRC05:40
*** tqtran has joined #openstack-infra05:44
*** sree has quit IRC05:45
*** sree has joined #openstack-infra05:46
*** tqtran has quit IRC05:49
*** wolverineav has joined #openstack-infra05:49
*** vsaienko has quit IRC05:49
*** rbuzatu has joined #openstack-infra05:50
*** sflanigan has joined #openstack-infra05:52
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job  https://review.openstack.org/42686305:53
*** amotoki has quit IRC05:53
*** dingyichen has joined #openstack-infra05:53
*** wolverineav has quit IRC05:53
*** krtaylor has joined #openstack-infra05:55
*** gyee has quit IRC05:58
*** ramishra_ has quit IRC05:58
*** ramishra has joined #openstack-infra06:00
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job with RabbitMQ  https://review.openstack.org/42630706:00
*** hamzy has quit IRC06:01
*** amotoki has joined #openstack-infra06:02
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job with RabbitMQ  https://review.openstack.org/42630706:04
*** nadya has joined #openstack-infra06:06
*** amotoki has quit IRC06:10
*** salv-orlando has joined #openstack-infra06:10
openstackgerritTomasz Trębski proposed openstack-infra/project-config master: Enable coverage jobs for monasca projects  https://review.openstack.org/43284506:11
*** pgadiya has joined #openstack-infra06:13
*** rbuzatu has quit IRC06:13
*** sree has quit IRC06:14
*** pcaruana has joined #openstack-infra06:20
*** amotoki has joined #openstack-infra06:21
*** krtaylor has quit IRC06:24
*** hamzy has joined #openstack-infra06:24
*** claudiub has joined #openstack-infra06:30
*** Jeffrey4l__ has joined #openstack-infra06:30
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix missing mutex release when aborting builds  https://review.openstack.org/43221106:31
*** psachin has joined #openstack-infra06:32
*** Jeffrey4l_ has quit IRC06:34
*** e0ne has joined #openstack-infra06:36
*** amotoki_ has joined #openstack-infra06:37
*** thorst_ has joined #openstack-infra06:39
*** krtaylor has joined #openstack-infra06:40
*** amotoki has quit IRC06:40
*** pahuang has quit IRC06:42
*** eliqiao has quit IRC06:42
*** thorst_ has quit IRC06:43
*** eliqiao has joined #openstack-infra06:46
*** aviau has quit IRC06:47
*** aviau has joined #openstack-infra06:48
*** skraynev has quit IRC06:51
*** rossella_s has quit IRC06:53
*** Swami__ has quit IRC06:53
*** Swami has quit IRC06:53
*** rossella_s has joined #openstack-infra06:53
*** nadya has quit IRC06:54
*** jascott1_ has joined #openstack-infra06:55
*** pahuang has joined #openstack-infra06:55
*** admcleod_ has quit IRC06:57
*** jascott1_ has quit IRC07:00
*** salv-orl_ has joined #openstack-infra07:01
*** salv-orlando has quit IRC07:03
*** anilvenkata has joined #openstack-infra07:04
*** rcernin has joined #openstack-infra07:07
openstackgerritHiroaki Kobayashi proposed openstack-infra/project-config master: Add release notes jobs for blazar  https://review.openstack.org/43127407:07
*** skraynev has joined #openstack-infra07:08
AJaegerianw, fungi: Searching is somehow broken - and was before. NO need for a revert. Let me send an email to the list. And check the json file...07:09
AJaegermmh, https://docs.openstack.org/user-guide/_static/js/ shows doc-characters-json07:11
AJaegerWhy common/js ?07:11
ianwAJaeger: cool, i thought so, didn't seem ssl related07:11
*** kaisers has joined #openstack-infra07:12
*** psachin has quit IRC07:13
*** rbuzatu has joined #openstack-infra07:14
ianwAJaeger: also, i've got a bit of a plan to make coverage graphs ... i'm thinking something like getting pbr to generate coverage xml reports (https://review.openstack.org/#/c/434071/) and then a "fake" test to get that into subunit output.07:15
ianwthat should get results into subunit2sql at which point openstack-health could query them07:15
ianwif that was run from post jobs, you'd get a moving view of code coverage07:15
ianwthat's the theory, anyway, i'll see how i go with it07:15
*** zz_dimtruck is now known as dimtruck07:17
AJaegerianw: interesting idea, look forward to it.07:20
*** jascott1_ has joined #openstack-infra07:20
*** markus_z has joined #openstack-infra07:23
*** isaacb has joined #openstack-infra07:25
*** jtomasek has joined #openstack-infra07:27
*** monester has joined #openstack-infra07:27
*** jtomasek_ has joined #openstack-infra07:27
*** dimtruck is now known as zz_dimtruck07:27
*** andreas_s has joined #openstack-infra07:28
*** e0ne has quit IRC07:29
*** knangia has quit IRC07:30
*** jascott1_ has quit IRC07:31
*** jascott1_ has joined #openstack-infra07:31
*** tesseract has joined #openstack-infra07:31
*** jtomasek has quit IRC07:32
*** jascott1_ has quit IRC07:35
*** yamamoto has quit IRC07:38
*** saibarspeis has joined #openstack-infra07:39
*** jtomasek_ is now known as jtomasek07:45
*** tqtran has joined #openstack-infra07:46
*** rbuzatu has quit IRC07:48
yolandagood morning07:49
*** tqtran has quit IRC07:50
*** isaacb has quit IRC07:51
*** rbuzatu has joined #openstack-infra07:52
*** rbuzatu has quit IRC07:56
*** yuval has joined #openstack-infra07:57
*** anilvenkata has quit IRC07:59
*** isaacb has joined #openstack-infra08:00
*** monester has quit IRC08:01
*** salv-orl_ has quit IRC08:10
*** Hal has joined #openstack-infra08:10
*** salv-orlando has joined #openstack-infra08:10
*** nmathew has quit IRC08:10
*** amotoki_ has quit IRC08:14
openstackgerritIsaac Beckman proposed openstack-infra/puppet-openstackci master: Pass Java arguments to Jenkins master  https://review.openstack.org/26302108:14
*** sree has joined #openstack-infra08:15
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/43413008:15
*** anilvenkata has joined #openstack-infra08:15
*** zz_dimtruck is now known as dimtruck08:18
*** sree has quit IRC08:19
*** aviau has quit IRC08:19
*** aviau has joined #openstack-infra08:19
*** ralonsoh has joined #openstack-infra08:20
*** ccamacho|Zzz is now known as ccamacho08:21
*** amotoki has joined #openstack-infra08:24
*** shardy has joined #openstack-infra08:24
*** martinkopec has joined #openstack-infra08:25
*** nadya has joined #openstack-infra08:27
*** nadya has quit IRC08:27
*** nadya has joined #openstack-infra08:27
*** dimtruck is now known as zz_dimtruck08:28
*** nmathew has joined #openstack-infra08:32
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix quickstart issues  https://review.openstack.org/42545908:32
*** sshnaidm|off is now known as sshnaidm08:32
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix quickstart issues  https://review.openstack.org/42545908:34
*** tuanluong has joined #openstack-infra08:35
*** esikachev has joined #openstack-infra08:36
*** links has quit IRC08:36
*** matrohon has joined #openstack-infra08:37
*** csomerville has quit IRC08:38
*** e0ne has joined #openstack-infra08:38
*** cody-somerville has joined #openstack-infra08:38
*** thorst_ has joined #openstack-infra08:40
openstackgerritMerged openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/43413008:41
*** dingyichen has quit IRC08:42
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job  https://review.openstack.org/42686308:42
*** dchen has quit IRC08:42
*** yonglihe has joined #openstack-infra08:42
*** matrohon has quit IRC08:43
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Compress logs to tar.gz archive  https://review.openstack.org/43414308:43
*** thorst_ has quit IRC08:44
*** yuanying has quit IRC08:47
*** yuanying has joined #openstack-infra08:47
*** yuanying has quit IRC08:47
*** jpich has joined #openstack-infra08:48
*** priteau has joined #openstack-infra08:49
*** matrohon has joined #openstack-infra08:52
*** links has joined #openstack-infra08:53
*** jascott1_ has joined #openstack-infra08:54
*** admcleod has joined #openstack-infra08:58
*** jascott1_ has quit IRC08:59
*** zzzeek has quit IRC09:00
*** zzzeek has joined #openstack-infra09:00
*** amotoki has quit IRC09:05
*** radez has quit IRC09:05
*** hashar has joined #openstack-infra09:08
*** lucas-afk is now known as lucasagomes09:10
*** panda|zZ is now known as panda09:11
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin)  https://review.openstack.org/42748209:13
openstackgerritMerged openstack-infra/project-config master: telemetry: run tripleo/scenario001 jobs as -nv  https://review.openstack.org/43373409:14
*** cmurphy has quit IRC09:16
*** cmurphy has joined #openstack-infra09:16
*** bhavik1 has joined #openstack-infra09:16
openstackgerritMerged openstack-infra/project-config master: Remove Puppet OpenStack beakers jobs on Ubuntu Trusty  https://review.openstack.org/43377509:17
openstackgerritMerged openstack-infra/project-config master: Add Install Guide Jobs to Designate  https://review.openstack.org/34167109:18
*** zz_dimtruck is now known as dimtruck09:19
openstackgerritMerged openstack-infra/project-config master: mistral: switch tripleo job to scenario003  https://review.openstack.org/43375409:20
*** rbuzatu has joined #openstack-infra09:20
openstackgerritMerged openstack-infra/project-config master: Move log-api tempest gate to voting stage  https://review.openstack.org/43301809:21
*** psachin has joined #openstack-infra09:21
*** yamamoto has joined #openstack-infra09:23
*** salv-orlando has quit IRC09:23
*** mhickey has joined #openstack-infra09:24
*** yamamoto_ has joined #openstack-infra09:24
*** e0ne has quit IRC09:24
*** aarefiev_afk is now known as aarefiev09:25
*** askb has quit IRC09:25
*** yamamoto_ has quit IRC09:25
*** yamamoto_ has joined #openstack-infra09:26
*** yamamoto_ has quit IRC09:26
*** links has quit IRC09:26
*** yuanying has joined #openstack-infra09:26
*** gildub has joined #openstack-infra09:27
*** yamamoto has quit IRC09:28
openstackgerritMerged openstack-infra/project-config master: Don't run nova dvsm tests for .git repo changes  https://review.openstack.org/36693309:28
*** dimtruck is now known as zz_dimtruck09:29
*** gcb has quit IRC09:31
*** dtantsur|afk is now known as dtantsur09:33
*** ssbarnea has joined #openstack-infra09:33
*** kzaitsev_mb has joined #openstack-infra09:33
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh  https://review.openstack.org/43102909:35
*** nmathew has quit IRC09:39
*** nmathew has joined #openstack-infra09:39
*** rfolco has joined #openstack-infra09:40
*** e0ne has joined #openstack-infra09:40
*** links has joined #openstack-infra09:43
*** Julien-zte has quit IRC09:50
*** wolverineav has joined #openstack-infra09:50
*** wolverin_ has joined #openstack-infra09:51
*** wolverineav has quit IRC09:51
*** Zara_ is now known as Zara09:51
*** jordanP has joined #openstack-infra09:54
*** salv-orlando has joined #openstack-infra09:54
*** nadya has quit IRC09:59
*** anilvenkata has quit IRC10:01
*** john-davidge has joined #openstack-infra10:01
*** hyakuhei has quit IRC10:01
*** hyakuhei has joined #openstack-infra10:01
*** hyakuhei has quit IRC10:01
*** hyakuhei has joined #openstack-infra10:01
*** rzetikx has joined #openstack-infra10:03
rzetikxHi, I would like to ask you what kind of statistics graph you are using for zuul status. I just know Grafana at the moment. Thanks10:06
openstackgerritJeremy Liu proposed openstack-infra/project-config master: Setup translation job for cloudkitty-dashboard  https://review.openstack.org/43419210:08
*** dtantsur is now known as dtantsur|bbl10:10
*** rbuzatu has quit IRC10:14
*** isaacb has quit IRC10:14
*** tuanluong has quit IRC10:14
*** isaacb has joined #openstack-infra10:15
*** rbuzatu has joined #openstack-infra10:15
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Move OSA upgrade job from experimental queue to periodic  https://review.openstack.org/41951710:18
odyssey4meAJaeger yolanda ^ For that patch, what I'm failing to see is where I define the branches and times that the periodic job should execute?10:19
odyssey4meIt looks like the timings are defined in the pipeline spec?10:19
*** mptacekx has joined #openstack-infra10:19
*** sambetts|afk is now known as sambetts10:19
*** denisra has joined #openstack-infra10:20
*** zz_dimtruck is now known as dimtruck10:20
*** yamahata has quit IRC10:20
*** liujiong has quit IRC10:21
*** isaacb has quit IRC10:25
*** denisra_ has joined #openstack-infra10:26
*** ociuhandu has joined #openstack-infra10:28
*** denisra has quit IRC10:30
*** dimtruck is now known as zz_dimtruck10:30
*** salv-orlando has quit IRC10:31
*** bhavik1 has quit IRC10:33
*** yanyanhu has quit IRC10:34
*** nadya has joined #openstack-infra10:37
*** thorst_ has joined #openstack-infra10:41
*** ldnunes has joined #openstack-infra10:41
*** nadya has quit IRC10:42
*** gongysh has quit IRC10:42
*** tosky has joined #openstack-infra10:43
*** wolverin_ has quit IRC10:43
*** wolverineav has joined #openstack-infra10:44
*** thorst_ has quit IRC10:46
*** masayukig_ is now known as masayukig10:49
*** masayukig is now known as masayukig_10:50
*** nadya has joined #openstack-infra10:50
*** masayukig_ is now known as masayukig10:50
*** ssbarnea has quit IRC10:52
*** wolverineav has quit IRC10:52
*** rossella_s has quit IRC10:53
*** rossella_s has joined #openstack-infra10:53
*** martinkopec has quit IRC10:55
*** ssbarnea has joined #openstack-infra10:56
*** nmathew has quit IRC10:56
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh  https://review.openstack.org/43102910:56
*** rbuzatu has quit IRC10:57
*** martinkopec has joined #openstack-infra10:58
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh  https://review.openstack.org/43102911:00
*** nicolasbock has joined #openstack-infra11:02
*** dtantsur|bbl is now known as dtantsur11:05
AJaegerodyssey4me: see how other periodic jobs are defined, You need to set the branch explicitely. Sorry, no time today to give exact instructions11:10
odyssey4meI'll try to puzzle it out again later, thanks AJaeger11:11
*** salv-orlando has joined #openstack-infra11:13
AJaegerodyssey4me: https://review.openstack.org/#/c/426308/1/jenkins/jobs/projects.yaml  - see the branch-override there as one example.11:15
AJaegerbbl11:15
odyssey4meawesome, thanks AJaeger11:15
openstackgerritGraham Hayes proposed openstack-infra/project-config master: Notify #openstack-dns of stable/ocata changes  https://review.openstack.org/43422111:16
*** masayukig is now known as masayukig_11:20
*** zz_dimtruck is now known as dimtruck11:22
*** Jeffrey4l__ is now known as Jeffrey4l11:22
*** isaacb has joined #openstack-infra11:22
*** wolverineav has joined #openstack-infra11:23
*** smarcet has joined #openstack-infra11:25
*** wolverineav has quit IRC11:28
*** dimtruck is now known as zz_dimtruck11:31
*** martinkopec has quit IRC11:32
*** martinkopec has joined #openstack-infra11:32
*** sdague has joined #openstack-infra11:34
*** jkilpatr has quit IRC11:35
openstackgerritJoshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Merge branch 'master' into feature/zuulv3  https://review.openstack.org/43423211:35
*** ldnunes has quit IRC11:36
*** ldnunes has joined #openstack-infra11:37
*** rcrit has quit IRC11:38
*** ociuhandu has quit IRC11:42
AJaegerodyssey4me: but that only works with devstack-gate jobs11:45
AJaegerodyssey4me: for others, see in project-config e.g. jenkins/jobs/python-bitrot-jobs.yaml11:46
AJaegerthe zuul-git-branch-prep-upper-constraints is the magic - use branch: stable/X to select the stable/X branch11:47
*** salv-orlando has quit IRC11:48
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job  https://review.openstack.org/42748111:49
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin)  https://review.openstack.org/42748211:50
*** yamamoto has joined #openstack-infra11:53
openstackgerritAndrea Frittoli proposed openstack-infra/elastic-recheck master: Add query for OOM issue on RAX 1664953  https://review.openstack.org/43423811:53
*** jkilpatr has joined #openstack-infra11:53
*** mdorman has quit IRC11:53
*** mdorman has joined #openstack-infra11:55
*** ociuhandu has joined #openstack-infra11:55
*** thiagolib has joined #openstack-infra11:57
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix quickstart issues  https://review.openstack.org/42545912:00
*** rbuzatu has joined #openstack-infra12:00
*** gildub has quit IRC12:01
*** raildo has joined #openstack-infra12:02
*** lucasagomes is now known as lucas-hungry12:04
*** sree has joined #openstack-infra12:04
*** wolverineav has joined #openstack-infra12:06
*** ldnunes has quit IRC12:07
*** rbuzatu has quit IRC12:08
openstackgerritSteven Hardy proposed openstack-infra/project-config master: tripleo-ci move multinode-upgrades-nv to check queue  https://review.openstack.org/43424712:09
*** wolverineav has quit IRC12:10
*** isaacb has quit IRC12:11
*** salv-orlando has joined #openstack-infra12:11
*** Mmike has left #openstack-infra12:12
*** masayukig_ is now known as masayukig12:19
*** rbuzatu has joined #openstack-infra12:19
*** ldnunes has joined #openstack-infra12:21
*** zz_dimtruck is now known as dimtruck12:21
*** sree has quit IRC12:24
*** hashar has quit IRC12:25
*** Qiming has quit IRC12:28
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci master: Add converge update to major upgrade test  https://review.openstack.org/43425012:29
*** Qiming has joined #openstack-infra12:30
*** dimtruck is now known as zz_dimtruck12:31
*** ricardoas has joined #openstack-infra12:34
openstackgerritJoshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Merge branch 'master' into feature/zuulv3  https://review.openstack.org/43423212:35
*** thorst_ has joined #openstack-infra12:38
*** faizy has quit IRC12:39
*** Julien-zte has joined #openstack-infra12:42
*** yamamoto has quit IRC12:43
*** panda is now known as panda|lunch12:44
*** wolverineav has joined #openstack-infra12:45
*** Julien-zte has quit IRC12:47
*** Julien-z_ has joined #openstack-infra12:47
*** ociuhandu has quit IRC12:48
*** wolverineav has quit IRC12:49
*** yamamoto has joined #openstack-infra12:50
*** yamamoto has quit IRC12:51
*** coreycb` is now known as coreycb12:51
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix quickstart issues  https://review.openstack.org/42545912:52
*** coreycb has quit IRC12:52
*** eliqiao has quit IRC12:52
*** coreycb has joined #openstack-infra12:53
*** fandi has joined #openstack-infra12:53
*** eliqiao has joined #openstack-infra12:55
*** smarcet has quit IRC12:58
*** hashar has joined #openstack-infra12:58
*** EricGonczer_ has joined #openstack-infra13:01
*** isaacb has joined #openstack-infra13:02
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove TODO comment that seems to be done  https://review.openstack.org/43396113:03
*** nadya has quit IRC13:03
*** trown|outtypewww is now known as trown13:04
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Re-enable test_footer_message  https://review.openstack.org/43048613:04
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Drop test_node_label  https://review.openstack.org/43047313:04
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Re-enable test_client_enqueue_ref test  https://review.openstack.org/39388713:05
*** slagle_ is now known as slagle13:06
*** kzaitsev_mb has quit IRC13:06
*** kzaitsev_mb has joined #openstack-infra13:07
*** rbuzatu has quit IRC13:08
*** lucas-hungry is now known as lucasagomes13:09
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job  https://review.openstack.org/43164913:10
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup  https://review.openstack.org/43373613:10
*** amotoki has joined #openstack-infra13:10
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job  https://review.openstack.org/43164913:13
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup  https://review.openstack.org/43373613:13
*** gouthamr has joined #openstack-infra13:14
*** eantyshev has quit IRC13:14
*** pgadiya has quit IRC13:14
*** faizy has joined #openstack-infra13:15
*** wolverineav has joined #openstack-infra13:15
*** jcoufal has joined #openstack-infra13:18
*** wolverineav has quit IRC13:20
*** zz_dimtruck is now known as dimtruck13:22
*** eharney has joined #openstack-infra13:26
*** panda|lunch is now known as panda13:27
*** jheroux has joined #openstack-infra13:28
*** rbuzatu has joined #openstack-infra13:28
*** eantyshev has joined #openstack-infra13:28
*** gordc has joined #openstack-infra13:29
*** dimtruck is now known as zz_dimtruck13:32
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job  https://review.openstack.org/43164913:32
*** salv-orlando has quit IRC13:34
openstackgerritMonty Taylor proposed openstack-infra/shade master: Wait for volumes to detach before deleting them  https://review.openstack.org/43277013:35
*** EricGonczer_ has quit IRC13:35
*** bswartz has joined #openstack-infra13:39
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job  https://review.openstack.org/43164913:39
*** rlandy has joined #openstack-infra13:40
*** wolverineav has joined #openstack-infra13:43
*** fandi has quit IRC13:45
*** rhallisey has joined #openstack-infra13:46
*** fguillot has joined #openstack-infra13:46
*** wolverineav has quit IRC13:47
*** ociuhandu has joined #openstack-infra13:51
*** rhallisey has quit IRC13:52
openstackgerritMonty Taylor proposed openstack-infra/jeepyb master: Move make_local_copy to utils  https://review.openstack.org/43384813:53
*** tongli has joined #openstack-infra13:54
pabelangerjianghuaw: yes, zookeeper is a hard dependency for nodepool and zuulv3 now13:55
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job  https://review.openstack.org/43164913:56
*** nadya has joined #openstack-infra13:57
*** wolverineav has joined #openstack-infra14:03
*** ihrachys has joined #openstack-infra14:04
*** isaacb has quit IRC14:05
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Set Node image_id and launcher attributes  https://review.openstack.org/43324214:05
openstackgerritMonty Taylor proposed openstack/os-client-config master: Add helper scripts to print version discovery info  https://review.openstack.org/43172014:05
*** isaacb has joined #openstack-infra14:06
*** cshastri has quit IRC14:07
*** Goneri has joined #openstack-infra14:08
*** rhallisey has joined #openstack-infra14:08
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Add generator API method for node iteration  https://review.openstack.org/43325214:08
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Disconnect from ZooKeeper at shutdown  https://review.openstack.org/43391914:08
*** wolverineav has quit IRC14:08
*** trown is now known as trown|brb14:08
*** mdrabe has joined #openstack-infra14:09
*** kgiusti has joined #openstack-infra14:10
*** jcoufal_ has joined #openstack-infra14:10
*** jcoufal has quit IRC14:11
jianghuawpabelanger, thanks. Could you suggest the reference doc for zookeeper setup?14:12
*** jcoufal has joined #openstack-infra14:12
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix missing mutex release when aborting builds  https://review.openstack.org/43221114:13
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Remove subnodes from nodepool  https://review.openstack.org/43240314:13
*** wolverineav has joined #openstack-infra14:13
*** jcoufal_ has quit IRC14:14
*** matt-borland has joined #openstack-infra14:15
*** jamesdenton has joined #openstack-infra14:15
pabelangerjianghuaw: we are currently using v0.5.5 of https://github.com/deric/puppet-zookeeper to configure zookeeper14:16
*** esikachev has quit IRC14:17
*** wolverineav has quit IRC14:17
*** salv-orlando has joined #openstack-infra14:18
*** salv-orlando has quit IRC14:18
*** salv-orlando has joined #openstack-infra14:18
jianghuawpabelanger, Beside of zookeeper, is there any update with zmq-publishers?14:19
jianghuawI redeployed a CI env which was setup one year ago. But after the redeployment, it seems the zmq-publisher not running well.14:19
jianghuawthe default port 8888 is not listened actually.14:19
openstackgerritMerged openstack-infra/shade master: Add accessor method to pull URLs from the catalog  https://review.openstack.org/43235814:20
pabelangermriedem:14:21
pabelangermriedem: sorry14:21
pabelangerjianghuaw: next version of nodepool will remove zmq-publishers, it will be 100% zookeeper moving forward14:21
openstackgerritPaul Carver proposed openstack-dev/hacking master: Disambiguate the hacking rules for third party imports  https://review.openstack.org/43430614:22
*** jamesdenton has quit IRC14:22
*** jsheeren has joined #openstack-infra14:22
*** zz_dimtruck is now known as dimtruck14:23
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job  https://review.openstack.org/43164914:24
jianghuawpabelanger, ah. The information is important. thanks.14:24
*** trown|brb is now known as trown14:24
*** jamesdenton has joined #openstack-infra14:25
jianghuawI've installed nodepool 0.3.1 in my env, could it fully work if I configure nodepool with zookeeper by without zmq-publishers?14:26
*** john-davidge has quit IRC14:26
*** john-davidge has joined #openstack-infra14:27
*** dprince has joined #openstack-infra14:27
*** dmellado_ is now known as dmellado14:27
pabelangerjianghuaw: no, 0.3.1 will still need zmq-publishers, out next releases (I don't know the version number ATM) will drop support.14:29
pabelangerjianghuaw: it is the feature/zuulv3 branch in git, if you want to review the code14:29
Shrewsjianghuaw: nodepool-builder is the only piece that uses ZooKeeper in 0.3.1. nodepoold still needs zmq14:29
pabelangerjianghuaw: we also have #zuul now, a dedicated channel for development, if you want to idle there too14:29
*** Krenair has quit IRC14:30
*** baoli has joined #openstack-infra14:30
*** links has quit IRC14:31
*** mriedem has quit IRC14:31
jianghuawpabelanger and Shrews: Ok. So the test node scheduling for jobs still depends on zmq?14:32
*** Julien-z_ has quit IRC14:32
*** Julien-zte has joined #openstack-infra14:32
*** dimtruck is now known as zz_dimtruck14:33
*** wolverineav has joined #openstack-infra14:33
AJaegerpabelanger: could you review https://review.openstack.org/#/c/432436/ , please? dependencies merged for fuel-web python-db14:33
pabelangerjianghuaw: right14:33
Shrewsyeah. and i guess technically nodepoold still needs zookeeper to get at image info, too14:33
Shrewsbut zmq and gearman are both going away in the next version14:34
mordredand mysql14:35
jianghuawShrews, does the nest version mean 0.4 or the following new version?14:35
jianghuaws/nest/next/14:35
Shrewsjianghuaw: not sure14:35
*** Krenair has joined #openstack-infra14:37
*** chlong has quit IRC14:37
pabelangerI thought jeblair was saying 1.0... but not 100%14:37
*** wolverineav has quit IRC14:37
jianghuawShrews and pabelanger, Thanks for your replying and explanation.14:39
*** tongli has quit IRC14:40
jianghuawShrews and pabelanger, Could you suggest any reference doc(if any) for me to make nodepool-0.3.1 working in my env easier? :-)14:40
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add reporter for Federated Message Bus (fedmsg)  https://review.openstack.org/42686114:43
Shrewsjianghuaw: oh, 0.4.0 is the zookeeper release. see http://lists.openstack.org/pipermail/openstack-infra/2017-January/005037.html14:43
openstackgerritKyrylo Shatskyy proposed openstack-infra/jenkins-job-builder master: Add support for 'Groovy Label Assignment Plugin'  https://review.openstack.org/42841914:43
dhellmanndid we have a zuul or gerrit blip earlier today? I have a patch that is approved and has run its gate jobs but that is not merged. It has no dependencies and doesn't seem to be tied to any other commits. https://review.openstack.org/#/c/433767/14:43
*** wolverineav has joined #openstack-infra14:44
dhellmannlast update about 40 minutes ago14:44
dhellmannfungi : ^^14:44
*** Swami has joined #openstack-infra14:44
jianghuawShrews, cool. thanks.14:45
andreykurilinhi folks! Can anyone explain me how I can add here custom dashboards https://review.openstack.org/#/admin/projects/openstack/rally,dashboards ?14:48
andreykurilinIt is possible?14:48
*** ociuhandu has quit IRC14:51
*** jcoufal_ has joined #openstack-infra14:51
*** esikachev has joined #openstack-infra14:52
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job  https://review.openstack.org/43164914:52
*** rossella_s has quit IRC14:53
*** hongbin has joined #openstack-infra14:53
pabelangerdhellmann: it looks fine now? It is possible zuul was still processing events14:53
*** tphummel has joined #openstack-infra14:53
*** rossella_s has joined #openstack-infra14:54
*** jcoufal has quit IRC14:54
dhellmannpabelanger : we've approved others since then and they merged, so I was worried about an issue. I added another W+1 to it and it is regating, which seems like an odd response since there was already a +2 from Jenkins, but maybe that's normal.14:56
pabelangerdhellmann: Ya, it is possible there might have been a network blip.  I can go look into the logs and see what happened14:57
dhellmannpabelanger : cool, thanks. Don't put a ton of time into it, I guess unless it's part of a bigger pattern.14:57
* dhellmann steps afk for a bit14:57
pabelangerack14:57
*** wolverin_ has joined #openstack-infra14:58
*** isaacb has quit IRC14:59
*** mriedem has joined #openstack-infra14:59
*** adrian_otto has joined #openstack-infra15:00
openstackgerritJeremy Liu proposed openstack-infra/project-config master: Setup translation job for cloudkitty-dashboard  https://review.openstack.org/43419215:00
*** wolverineav has quit IRC15:01
*** zz_dimtruck is now known as dimtruck15:03
*** rbrndt has joined #openstack-infra15:04
*** jamesdenton has quit IRC15:05
AJaegerpabelanger, mordred : could you review https://review.openstack.org/#/c/432436/ , please? dependencies merged for fuel-web python-db change15:06
*** jamesdenton has joined #openstack-infra15:07
fungipabelanger: dhellmann: yeah, i hope we've got something in the main zuul log from around 14:06 indicating why it didn't manage to merge it in gerrit and yet also didn't leave a comment saying so15:07
*** jaosorior has quit IRC15:09
fungiaha, raised an exception. i'll put the tb on paste.o.o in a sec15:10
*** jaosorior has joined #openstack-infra15:10
*** eroux has quit IRC15:10
*** isaacb has joined #openstack-infra15:10
*** lejit has quit IRC15:13
fungidhellmann: pabelanger: jeblair: http://paste.openstack.org/show/599050/ coupled with the fact that the succeeded comment and verify +2 appeared on https://review.openstack.org/433767 at 14:06 suggests that zuul hit a 2-minute timeout waiting on a response from gerrit's api... do you concur?15:15
fungioh, and ignore that stray logline about the ironic job. it's unrelated but i missed trimming it from my paste15:16
*** jamesden_ has joined #openstack-infra15:16
*** mtanino has joined #openstack-infra15:16
fungii'm digging in the gerrit error log now to see if i can spot any relevant needles in the haystack at that end15:17
*** jamesdenton has quit IRC15:18
*** EricGonczer_ has joined #openstack-infra15:18
*** armax has joined #openstack-infra15:18
*** burgerk has joined #openstack-infra15:19
*** eroux has joined #openstack-infra15:19
fungiick, looks like db communication issues? http://paste.openstack.org/show/599052/15:21
pabelangerfungi: thanks for looking, got pulled away15:22
mordredfungi: wasn't there some email about something trove something something for rackspace?15:22
mordreddid we maybe just hit that?15:22
fungiso looks more like zuul did probably get an error back, and the two-minute timeout was probably on gerrit's end trying to perform a database lookup15:22
fungimordred: that's the next place i was going to look ;)15:22
*** saibarspeis has quit IRC15:22
*** andymaier has joined #openstack-infra15:23
*** claudiub|2 has joined #openstack-infra15:23
*** radez has joined #openstack-infra15:23
*** jsheeren has quit IRC15:24
*** saibarspeis has joined #openstack-infra15:24
fungino alerts or maintenance for https://status.rackspace.com/ and no trove-related trouble tickets for that tenant. pending maintenances we're being alerted to are host maintenance which will impact elasticsearch02 on the 19th and block storage maintenance impacting static and wiki-dev on the 28th15:25
*** claudiub has quit IRC15:26
fungino unusual spikes on the cacti graphs for review.o.o around that timeframe either15:27
*** adrian_otto has quit IRC15:27
*** adrian_otto has joined #openstack-infra15:28
fungihttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=34&rra_id=all shows a little bit of an inbound network traffic spike on the trove-facing interface, so maybe that was it15:28
*** nadya has quit IRC15:28
fungibut it's barely a third the size of the spikes we get at db backup time15:29
fungii guess it could have been particularly unlucky timing15:29
*** mriedem1 has joined #openstack-infra15:31
dhellmannfungi, pabelanger : it seems likely that it was bad timing; I'll let you know if it shows up again15:31
fungithanks, dhellmann15:31
openstackgerritMonty Taylor proposed openstack/os-client-config master: Add support for overriding mistral service type  https://review.openstack.org/43434415:32
*** mriedem has quit IRC15:33
fungiprobably the best we could do for future-proofing is to either properly report back an error to gerrit and/or check the resulting state and retry outstanding operations. but this seems infrequent enough that can probably wait for zuul v3 (and zuul pushing its own merge commits probably makes this even less likely still)15:33
*** yolanda has quit IRC15:34
*** nadya has joined #openstack-infra15:35
jeblairfungi: well, the 'report back an error to gerrit' is tricky if gerrit can't update its db...15:35
*** yolanda has joined #openstack-infra15:35
fungijeblair: agreed15:36
fungithe catch-22 there dawned on me right after i said it15:36
*** mriedem1 is now known as mriedem15:37
jeblairi think maybe a 'try 3 times then give up' algorithm would be okay -- but part of the intent with the current code is not to let gerrit get zuul completely stuck.15:37
fungiapparently in this case the db problem didn't prevent zuul from commenting/voting so a followup error comment likely would have shown up15:37
jeblairah yeah, then even better for the try-3-times approach15:38
fungithough not necessarily the case of course, and i can certainly envision situations where whatever caused the action to fail would have also prevented followup comments15:38
fungibut yes, a likely improvement at least15:39
jeblairyep, like "this change is broken in the gerrit db"15:39
mordred"this change is cut in half"15:40
jeblairmordred: did you cut this change in half?15:40
fungieric the half a change15:41
mordredjeblair: this change is cut in half. I'd like to exchange it for one that is not ... cut in half15:41
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Re-enable devstack test job  https://review.openstack.org/43164915:44
*** mriedem has quit IRC15:52
*** amotoki has quit IRC15:55
*** sree has joined #openstack-infra15:56
*** sree has quit IRC15:56
*** abregman has joined #openstack-infra15:56
*** sree has joined #openstack-infra15:56
*** mriedem has joined #openstack-infra15:56
*** abregman has quit IRC15:56
*** Julien-zte has quit IRC15:57
*** abregman has joined #openstack-infra15:58
*** knangia has joined #openstack-infra15:58
*** jascott1_ has joined #openstack-infra15:58
*** psachin has quit IRC15:58
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node cleanup  https://review.openstack.org/43373616:00
*** vhosakot has joined #openstack-infra16:00
*** xarses_ has joined #openstack-infra16:00
*** kzaitsev_mb has quit IRC16:01
*** marst has joined #openstack-infra16:01
*** yuval has quit IRC16:01
*** andreww has joined #openstack-infra16:02
*** chlong has joined #openstack-infra16:04
*** ccamacho has left #openstack-infra16:04
*** nadya has quit IRC16:05
*** xarses_ has quit IRC16:06
*** xarses_ has joined #openstack-infra16:06
*** flepied has quit IRC16:06
*** andreww has quit IRC16:06
*** andreww has joined #openstack-infra16:07
*** nadya has joined #openstack-infra16:07
*** yamahata has joined #openstack-infra16:07
*** thorst_ has quit IRC16:08
*** ricardoas has quit IRC16:08
*** ihrachys has quit IRC16:08
*** thorst_ has joined #openstack-infra16:09
*** isaacb has quit IRC16:09
*** morgan_ is now known as morgan16:10
*** xarses_ has quit IRC16:10
*** xarses_ has joined #openstack-infra16:11
*** xarses_ has quit IRC16:11
*** andreww has quit IRC16:12
*** xarses_ has joined #openstack-infra16:12
*** erhudy has joined #openstack-infra16:14
*** erikmwilson has quit IRC16:15
*** nadya has quit IRC16:15
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable devstack test job  https://review.openstack.org/43164916:15
*** EricGonczer_ has quit IRC16:15
*** erikmwilson has joined #openstack-infra16:15
*** aarefiev is now known as aarefiev_afk16:15
*** andreww has joined #openstack-infra16:16
*** xarses_ has quit IRC16:20
*** nadya has joined #openstack-infra16:20
*** mptacekx has quit IRC16:21
*** flepied has joined #openstack-infra16:22
openstackgerritBéla Vancsics proposed openstack-dev/hacking master: Add a check to make sure the right assert* method is used  https://review.openstack.org/43436616:22
*** cinerama` is now known as cinerama16:23
*** saibarspeis has quit IRC16:24
*** rossella_s has quit IRC16:25
*** rzetikx has quit IRC16:25
*** EricGonczer_ has joined #openstack-infra16:25
AJaegerfungi, pabelanger, mordred : could you put https://review.openstack.org/#/c/432436/ on your review queue, please? dependencies merged for fuel-web python-db change...16:27
*** jcoufal has joined #openstack-infra16:28
pabelangerdone16:28
AJaegerthanks16:28
*** faizy has quit IRC16:28
*** baoli has quit IRC16:30
openstackgerritMonty Taylor proposed openstack/os-client-config master: Remove the keystoneclient auth fallback  https://review.openstack.org/43437216:30
*** baoli has joined #openstack-infra16:31
*** jcoufal_ has quit IRC16:31
*** jcoufal__ has joined #openstack-infra16:31
*** ihrachys has joined #openstack-infra16:32
*** jcoufal has quit IRC16:32
*** jcoufal_ has joined #openstack-infra16:33
*** annegentle has joined #openstack-infra16:33
*** vhosakot has quit IRC16:33
*** martinkopec has quit IRC16:34
*** jcoufal__ has quit IRC16:36
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh  https://review.openstack.org/43102916:36
*** sandanar has quit IRC16:36
*** annegentle has quit IRC16:37
*** annegentle has joined #openstack-infra16:38
*** thorst_ has quit IRC16:38
*** adrian_otto has quit IRC16:39
*** mdrabe has quit IRC16:39
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci master: Add converge update to major upgrade test  https://review.openstack.org/43425016:39
mriedemianw: did you find out if nodepool is running devstack on trusty? https://review.openstack.org/#/c/433274/16:39
fungimriedem: as discussed in our meeting yesterday, it's xenial16:40
fungi(also ianw is probably sound asleep by now)16:40
mriedemok good to know16:41
fungithe risk to us is that we're running nodepool on trusty servers and doing integration testing of changes to it on xenial so there's the potential for us to break production, but the solution to that is to upgrade our prod servers16:41
*** sree has quit IRC16:41
clarkbwhich is straightforward for the riskiest servers (image builders) and planned for post ptg now16:42
*** jcoufal has joined #openstack-infra16:42
*** terryw is now known as otherwiseguy16:42
*** sree has joined #openstack-infra16:42
openstackgerritKaitlin Farr proposed openstack-infra/project-config master: Enable signature verification for barbican-tempest-plugin gate  https://review.openstack.org/43438416:42
*** sree has quit IRC16:43
*** sree has joined #openstack-infra16:43
*** rcernin has quit IRC16:43
*** masayukig is now known as masayukig_16:45
*** masayukig_ is now known as masayukig16:45
clarkbpabelanger et al, are there any other items that I can help with for ptg prep? I got most of my TODOs sorted yesterday with translate-dev upgraded and elasticsearch green again. So let me know if I can help somewhere16:45
*** jcoufal_ has quit IRC16:45
clarkbotherwise will likely spend time prepping the how to debug test failures thing16:45
*** jaypipes has joined #openstack-infra16:45
*** tosky has quit IRC16:46
pabelangerclarkb: I was planning on starting nl01.o.o today, if you want to look at zuulv3-dev.o.o, I am okay with that16:49
*** jamesden_ has quit IRC16:49
*** tosky has joined #openstack-infra16:49
*** hieulq_ has joined #openstack-infra16:49
pabelangerclarkb: I was thinking of just standing up a plan server, so we can then iterate on zuulv3 patches as we land them16:49
*** abregman has quit IRC16:49
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job  https://review.openstack.org/42748116:49
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin)  https://review.openstack.org/42748216:49
*** thorst_ has joined #openstack-infra16:49
clarkbpabelanger: I reiviewed the zuulv3-dev patch too I found one issue with the ansible grouping16:49
clarkbpabelanger: but that was yesterday so maybe I need to go look at new patches?16:49
pabelangerclarkb: ya, I haven't uploaded anything new yet16:50
clarkbits the ansible grouping thing that always trips me up so now I know to look for it :)16:51
*** kfarr has joined #openstack-infra16:52
*** andymaier has quit IRC16:52
openstackgerritIan Cordasco proposed openstack-infra/project-config master: Add IRC notifications from Gerrit for #craton  https://review.openstack.org/43439116:53
*** tqtran has joined #openstack-infra16:55
sigmavirusHey, so #craton has a tox env that we run separately from the standard `py35` environment because it contains slower functional tests. Is there a good way to update our check/gate jobs to run that as well?16:56
* sigmavirus suspects we have to define a custom job for that to work, but was hoping that wasn't the case16:56
AJaegersigmavirus: there are many examples in project-config on how to do that ;)16:57
*** mdrabe has joined #openstack-infra16:57
AJaegersigmavirus: search for gate-{name}-tox-{envlist}-{node} in jenkins/jobs/projects.yaml16:57
AJaegerenvlist is your environment16:57
sigmavirusah16:58
AJaegersigmavirus: that template will run "tox -e envlist" for you16:58
sigmavirusI was looking in the wrong place16:58
sigmavirusThanks AJaeger !16:58
*** tesseract has quit IRC16:58
*** jamesdenton has joined #openstack-infra16:58
AJaegersigmavirus: and then add it to your repo in zuul/layout.yaml as well to trigger when to run16:58
*** jascott1_ has quit IRC16:58
*** adrian_otto has joined #openstack-infra16:59
*** jascott1_ has joined #openstack-infra16:59
*** ramishra has quit IRC17:01
*** jascott1_ has quit IRC17:03
*** trown is now known as trown|lunch17:04
*** jordanP has quit IRC17:04
*** erikmwilson has quit IRC17:05
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add lock state to node listing  https://review.openstack.org/43439717:05
openstackgerritIan Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate  https://review.openstack.org/43439917:06
*** dtantsur is now known as dtantsur|afk17:06
annegentleHuh. I coulda sworn there was a Launchpad for logging bugs against the www.openstack site. Where'd it go?17:06
AJaegerannegentle: https://bugs.launchpad.net/openstack-org  ?17:07
clarkbannegentle: openstack-org17:07
annegentlethere it is!17:07
annegentleLOL thanks to you both17:07
openstackgerritMonty Taylor proposed openstack-infra/shade master: Produce some debug details about nova usage objects  https://review.openstack.org/43440117:08
*** erikmwilson has joined #openstack-infra17:08
*** faizy has joined #openstack-infra17:09
*** tqtran has quit IRC17:11
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config master: grafana: track new rally py35 job for neutron  https://review.openstack.org/43440217:11
*** e0ne has quit IRC17:11
*** andreas_s has quit IRC17:11
clarkbAJaeger: wins with the proper url :)17:12
AJaegerclarkb: this is not about winning ;)17:13
annegentlesnerk17:14
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Re-enable test_diskimage_build_only test  https://review.openstack.org/43326517:14
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Re-enable test_dib_upload_fail test  https://review.openstack.org/43323517:14
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Re-enable test_image_upload_fail test  https://review.openstack.org/43327017:14
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Re-enable working test_builder.py tests  https://review.openstack.org/43326217:14
annegentleBug logged: https://bugs.launchpad.net/openstack-org/+bug/1665041 for a summit voting login problem17:14
openstackLaunchpad bug 1665041 in openstack-org "Vote for speakers link login seems broken" [Undecided,New]17:14
openstackgerritMonty Taylor proposed openstack-infra/shade master: Try to capture novaclient debug logging  https://review.openstack.org/43440417:14
*** d0ugal has quit IRC17:16
*** beekneemech is now known as bnemec17:16
*** csomerville has joined #openstack-infra17:21
*** priteau has quit IRC17:21
*** markus_z has quit IRC17:21
*** EricGonczer_ has quit IRC17:22
*** jamesdenton has quit IRC17:22
*** baoli has quit IRC17:22
*** jbernard_ is now known as jbernard17:23
*** jascott1_ has joined #openstack-infra17:24
*** jascott1_ has quit IRC17:24
*** chlong has quit IRC17:24
*** cody-somerville has quit IRC17:24
*** Apoorva has joined #openstack-infra17:25
*** annegentle has quit IRC17:25
openstackgerritIan Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate  https://review.openstack.org/43439917:25
*** EricGonczer_ has joined #openstack-infra17:28
*** d0ugal has joined #openstack-infra17:30
*** d0ugal has quit IRC17:30
*** d0ugal has joined #openstack-infra17:30
*** sabayonuser has joined #openstack-infra17:30
*** thiagolib has quit IRC17:31
*** ralonsoh has quit IRC17:31
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate master: WIP/DNM: Testing grenade with Ironic multi-node job  https://review.openstack.org/43177717:33
*** ZZelle_ has joined #openstack-infra17:33
*** jaosorior has quit IRC17:35
*** annegentle has joined #openstack-infra17:35
openstackgerritMonty Taylor proposed openstack-infra/shade master: Force nova microversion to 2.0  https://review.openstack.org/43441617:36
*** lucasagomes is now known as lucas-afk17:38
*** mriedem has quit IRC17:40
*** jpich has quit IRC17:40
*** baoli has joined #openstack-infra17:41
*** d0ugal has quit IRC17:42
*** faizy has quit IRC17:45
openstackgerritKaitlin Farr proposed openstack-infra/project-config master: Enable signature verification for barbican-tempest-plugin gate  https://review.openstack.org/43438417:47
kfarrsdague ping!  I have a question about setting a nova conf value in the gate17:47
kfarrYou had a few comments in https://review.openstack.org/#/c/383293/ about using post-config17:48
*** rossella_s has joined #openstack-infra17:48
kfarrand I think I configured it correctly here, but was hoping you could take a look: https://review.openstack.org/#/c/434384/17:48
*** mhickey has quit IRC17:51
*** yamahata has quit IRC17:51
*** hashar has quit IRC17:52
pabelangerjeblair: do you mind +3 430329 and 43033917:52
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add image list to debug output on failure  https://review.openstack.org/43442717:52
openstackgerritMonty Taylor proposed openstack-infra/shade master: Consume cirros qcow2 image if it's there  https://review.openstack.org/43442817:52
*** vhosakot has joined #openstack-infra17:53
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh  https://review.openstack.org/43102917:54
*** rossella_s has quit IRC17:54
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate master: Put neutron config at the top of the local.conf file  https://review.openstack.org/43443017:55
*** vsaienko has joined #openstack-infra17:56
*** jcoufal has quit IRC17:59
*** EricGonczer_ has quit IRC17:59
*** tphummel has quit IRC18:01
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck master: Add query for bug 1665053  https://review.openstack.org/43443618:01
openstackbug 1665053 in tempest "test_stamp_pattern times out waiting for the device to show up on the guest" [Undecided,New] https://launchpad.net/bugs/166505318:01
harlowjazxiiro hi there, do u mind looking over https://review.openstack.org/#/c/403940/18:01
harlowjait'd be nice to merge that so that at godaddy we can use it :)18:02
harlowjai don't want to go to the UI anymore to make this stuff, lol18:02
*** hieulq_ has quit IRC18:02
*** ihrachys_ has joined #openstack-infra18:02
*** ihrachys has quit IRC18:02
*** panda is now known as panda|bbl18:04
*** sambetts is now known as sambetts|afk18:04
openstackgerritVarun Gadiraju proposed openstack-infra/devstack-gate master: WIP/DNM: Testing grenade with Ironic multi-node job  https://review.openstack.org/43443918:04
*** tonytan4ever has joined #openstack-infra18:05
*** Goneri has quit IRC18:09
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci master: Remove test-environments/multinode_major_upgrade.yaml  https://review.openstack.org/42637518:12
*** tqtran has joined #openstack-infra18:13
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck master: Fix spaces in 2 queries  https://review.openstack.org/43444518:13
*** jamielennox is now known as jamielennox|away18:13
zxiiroharlowja: hey, I'm currently traveling at a conference. If you cc me on it, it will go into my mail queue which I'll get to as soon as I'm back from travel18:13
harlowjakk18:14
jeblairpabelanger: +318:14
*** kfarr has quit IRC18:14
*** Goneri has joined #openstack-infra18:14
*** esikachev has quit IRC18:15
pabelangerdanke18:15
Shrewsc 218:15
Shrewsugh18:16
pabelangerug ho18:16
pabelangerlogs.o.o is full again18:16
*** spzala has joined #openstack-infra18:16
pabelanger/dev/mapper/main-logs        12682583968 12682567584          0 100% /srv/static/logs18:16
* fungi sighs18:17
*** Swami_ has joined #openstack-infra18:17
bkeroagain?18:17
*** tphummel has joined #openstack-infra18:17
clarkblast time was due to many gigabyte job logs18:17
*** tqtran has quit IRC18:17
*** adrian_otto has quit IRC18:17
clarkbwe probably need to do an audit again and see if we have that going on18:17
jeblairhttp://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=717&rra_id=4&view_type=&graph_start=1454129464&graph_end=148718264818:17
fungicurrent retention is 60 days18:18
*** flepied has quit IRC18:18
*** esikachev has joined #openstack-infra18:18
fungiwe generate 12tb of test logs (compressed) in two months18:18
fungishall i shrink that to 45 days for now to get us some breathing room so people can see what's being logged too excessively?18:19
jeblairfungi: i like that plan18:19
pabelanger++18:19
mtreinishfungi: that's a lot of logs18:20
*** Swami has quit IRC18:20
jeblair(also, when we write the zuul v3 log copying playbook, maybe we should see about logging the size of the logs we copy somewhere, so we can more easily track down large jobs in the future)18:20
pabelangerYa, that is a good idea18:20
mtreinishfungi: you might want to put that on the ML too18:20
fungithere's a log_archive_maintenance.sh script which has been running for the past 4 days18:21
pabelangerstuck?18:21
jeblairProcess 20918 attached18:21
jeblairwait4(-1,18:21
fungistrace -p 25422 (the child process) looks active18:22
* fungi waits for his terminal to catch up with the sigint he sent18:22
*** esikachev has quit IRC18:22
*** jkilpatr has quit IRC18:22
jeblairyeah, it's waiting on find18:22
jeblairwhich is running18:22
fungii'm going to abort that since it's likely bogged down trying to do compression, and run a modified version which just handles deletion18:22
fungito get us back on track faster18:23
fungithen we can resume the normal one18:23
fungiany objections?18:23
jeblairwe don't know how long that script normally takes do we?18:23
fungican probably infer from cpu graphs in cacti?18:24
fungii have it queued up in a root screen session on static.o.o ready to go18:24
pabelangerI want to say over 24 hours the last time I ran it in screen18:24
jeblairfungi: no objection18:25
*** esikachev has joined #openstack-infra18:26
fungiprevious finds/maint script killed, modified one which just does deletions is running now18:27
*** jheroux has quit IRC18:27
fungi337M available already18:28
fungijobs should no longer be hemmoraging18:28
fungiheh, and then something got uploaded and we're back to 123M available ;)18:28
*** jkilpatr has joined #openstack-infra18:29
pabelangerclarkb: do you have a script you used to audit the logs folder?18:29
clarkbpabelanger: no I didn't do anything sophisicated just ran du18:30
pabelangerk18:30
clarkbpabelanger: I ran it a little bit into the fs tree so that it wasn't hitting everything but that was diverse enough to see oh wow these 3 changes are all multigigabyte18:30
clarkboh and they are all part of $project18:30
clarkband go from there18:30
fungilooks like we're slowly making traction, deleting more than is being uploaded18:30
* dims peeks18:31
*** esikachev has quit IRC18:31
*** tqtran has joined #openstack-infra18:31
* dims spots "<pabelanger>logs.o.o is full again"18:31
*** yamahata has joined #openstack-infra18:31
fungisix more weeks of winter?18:32
dimslol18:32
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate master: WIP/DNM: Testing grenade with Ironic multi-node job  https://review.openstack.org/43177718:33
jlvillalAnyone else noticing "POST_FAILURE" errors?18:34
*** flepied has joined #openstack-infra18:34
jlvillalIn Zuul18:34
pabelangeryes18:34
pabelangerlogs.o.o is full, we are deleting content18:34
jlvillalAh, thanks!18:34
*** ihrachys_ has quit IRC18:34
fungi(some?) jobs should be succeeding now18:35
*** adrian_otto has joined #openstack-infra18:35
fungii keep seeing us get up to nearly 1gb free and then drop back down to almost none18:35
*** ihrachys has joined #openstack-infra18:35
fungiyep, now we're at 100K available :/18:35
*** Swami_ has quit IRC18:37
*** dhinesh has joined #openstack-infra18:37
pabelangerI see we don't compress journal logs18:38
clarkbpabelanger: looks like I did something like: du -hs /srv/static/logs/35* | sort -h18:38
fungisomething tells me it's going to take some time for find to happen across enough 45-60 day old logs to delete18:38
pabelangerfound some in tripleo jobs, around 30MB uncompressed18:38
clarkbthen 36* eetc until I found some patterns then attacked those rather than bulk18:39
fungiright now it's not removing them fast enough to offset the incoming volume18:39
clarkbpabelanger: sorry 35/* I think18:39
fungii'm going to guess that the reason the log maintenance script was still running 4 days in was that up until yesterday it was competing with mlocate update scans18:40
clarkbpabelanger: 30MB uncompressed is still pretty small in grand scheme of things, previouslly it was single jobs with 2GB of logs18:40
*** adrian_otto has quit IRC18:40
pabelangerclarkb: ya, these are 250MB each, with multiple rechecks18:40
*** ihrachys has quit IRC18:41
*** mriedem has joined #openstack-infra18:42
*** esikachev has joined #openstack-infra18:42
*** ihrachys has joined #openstack-infra18:43
*** trown|lunch is now known as trown18:44
*** salv-orlando has quit IRC18:44
openstackgerritMonty Taylor proposed openstack-infra/shade master: Consume cirros qcow2 image if it's there  https://review.openstack.org/43442818:44
*** MasterOfBugs has joined #openstack-infra18:45
pabelangerclarkb: tripleo check jobs, are doing 603M right now.  Looking to see what is going on18:45
*** Goneri has quit IRC18:45
*** esikachev has quit IRC18:46
clarkbmy du is still being really slow :(18:47
fungiit's probably competing with my find -exec rm18:48
pabelangerclarkb: So, where do I update things to start compressing a new log file?18:48
fungicompeting for i/o bandwidth i mean18:48
openstackgerritIan Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate  https://review.openstack.org/43439918:48
clarkbpabelanger: ideally jobs would compress as much as sane, then we have our fallback cron that sweeps through and picks up the leftovers18:48
pabelangerk18:48
clarkbpabelanger: that distributes the load and reduces network bw needed to do copies18:49
fungiwhat filename pattern are these uncompressed files? maybe we can use this to make some quick traction18:49
*** zara_the_lemur__ has joined #openstack-infra18:49
pabelanger/srv/static/logs/01/422801/1/check/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/9018e9c/logs/var/log/journal/ef56f1b7283141aea559c3525915a76618:49
pabelangerfor example18:49
*** saibarspeis has joined #openstack-infra18:49
pabelangerare 40GB uncompressed18:49
fungiso logs/var/log/journal/*18:50
*** ankur-gupta-f4 has joined #openstack-infra18:50
pabelangerya18:50
fungiwhat are they?18:50
pabelangersystem.journal and user-3000.journal18:50
pabelangerthey are actually binary18:50
mordredsounds like systemd logs yeah?18:50
fungioic, those are directory names18:50
pabelangerso, not sure it makes sense to archive them?18:51
clarkbpabelanger: devstack-gate archives the compressed text output of journalctl instead fwiw18:51
*** sdague has quit IRC18:51
pabelangerclarkb: let me see how tripleo is doing it18:51
fungi`du -sh /srv/static/logs/01/422801/1/check/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/9018e9c/logs/var/log/journal/ef56f1b7283141aea559c3525915a766` says that's only 33mib18:52
funginot 40gib18:52
pabelangerfungi: typo on my side, I meant MB18:52
fungioh, that's tiny18:52
fungicomparatively speaking18:52
mordredyah18:52
pabelangermultiple rechecks18:52
mordred40GB of systemd logs is huge - and also I didn't blink at all thinking that might be the size of the logs :)18:52
pabelangerand multi nodes18:52
pabelangerso, it adds up18:53
openstackgerritVictor Morales proposed openstack-dev/pbr master: Add memory tracer argument  https://review.openstack.org/43394718:53
pabelanger1.2G /srv/static/logs/01/42280118:54
*** sdague has joined #openstack-infra18:55
*** jamesdenton has joined #openstack-infra18:55
fungiwhich is about 0.01% of the current volume18:55
openstackgerritMonty Taylor proposed openstack-infra/shade master: Consume cirros qcow2 image if it's there  https://review.openstack.org/43442818:55
fungiso not tiny, but for all jobs of all rechecks of a change it's not giant either18:56
pabelangerfungi: right, but other logs are only a few MB18:56
*** sindhu has joined #openstack-infra18:56
clarkbif my du ever finishes hopefully we can get a bit more insight at a higher level18:56
clarkbpabelanger: devstack-gate jobs were running inthe ~600MB range before iirc18:56
clarkbso I don't think thats true18:56
clarkb(but higher level du should confirm if it ever finishes)18:56
* fungi picks a random recent change for an example18:56
*** tosky has quit IRC18:57
pabelanger7.0G ./41460118:57
pabelangeranother tripleo job18:57
clarkbpabelanger: thats not a job thats a change18:57
pabelangermybad18:57
*** tnovacik has joined #openstack-infra18:57
clarkbso the change in total used 7GB which isn't tiny, but is there a specific job acting up? in the past it was literally one jobs made 2GB of logs18:58
*** esikachev has joined #openstack-infra18:58
clarkbthen run that job a bunch and single change could be using 100 GB18:58
pabelangerthis one is about 30 different patch uploads18:58
fungihrm... /srv/static/logs/16/434316/1/check/gate-grenade-dsvm-neutron-ubuntu-xenial/1ca37b1 ran for one hour and succeeded, du -sh says total volume is 26mib18:58
fungioh, wait, grenade18:58
pabelangerand multiple rechecks it seems18:58
clarkb(its good info regardless but important to distinguish between job cost vs total aggregate change cost)18:58
fungibut /srv/static/logs/16/434316/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7adca73 is 55mib18:59
clarkbok mabe it was 600MB uncompressed19:00
clarkband I have my numbers mixed up19:00
openstackgerritIan Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate  https://review.openstack.org/43439919:00
pabelangerya, with /srv/static/logs/01/414601/24/check/gate-tripleo-ci-centos-7-3nodes-multinode-nv/3012361 it is 101M, but 75MB of that is the journal logs19:00
fungiwe've made back >1gib of traction so maybe we're finally overcoming the inbound volume19:02
*** dharinic is now known as dharinic|lunch19:02
*** pcaruana has quit IRC19:03
*** esikachev has quit IRC19:04
clarkbI've restarted du with a smaller set of changes to hopefully get back something in a reasonable amount of time19:06
*** MasterOfBugs has quit IRC19:07
clarkbdoing a few one offs for changes ~300MB seems to be normalish19:07
*** MasterOfBugs has joined #openstack-infra19:07
clarkbbut thats with only a few patchsets and no rechecks looks like19:07
*** adrian_otto has joined #openstack-infra19:07
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci master: Exclude /var/log/journal from devstack-logs  https://review.openstack.org/43446719:08
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci master: Add converge update to major upgrade test  https://review.openstack.org/43425019:09
*** esikachev has joined #openstack-infra19:09
*** EricGonczer_ has joined #openstack-infra19:09
fungiwe've been consistently over 1gib available for the past 10 minutes or so now (just climbed to 2gib), so hopefully the worst is behind us for now19:09
*** shardy has quit IRC19:10
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck master: Add query for bug 1664793  https://review.openstack.org/43443619:11
openstackbug 1664793 in OpenStack Compute (nova) "test_stamp_pattern failing in ceph jobs" [Undecided,Confirmed] https://launchpad.net/bugs/166479319:11
openstackgerritIan Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate  https://review.openstack.org/43439919:11
*** ihrachys_ has joined #openstack-infra19:13
*** ihrachys has quit IRC19:15
*** esikachev has quit IRC19:17
openstackgerritMerged openstack-infra/elastic-recheck master: Fix spaces in 2 queries  https://review.openstack.org/43444519:18
openstackgerritIan Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate  https://review.openstack.org/43439919:18
*** EricGonczer_ has quit IRC19:18
clarkbI fail at du'ing current one which should'ev been relatively quick is still going. Maybe there is a massive change dir int here?19:19
clarkbpabelanger: fungi http://paste.openstack.org/show/599100/ one of these is not like the others19:21
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci master: Exclude /var/log/journal from devstack-logs  https://review.openstack.org/43446719:21
fungiyeah, we just lost our traction. back to 0 bytes available again19:22
bkero:'( logs19:22
*** sdake_ has joined #openstack-infra19:22
clarkbsuper preliminary data says kolla likely our problem19:22
clarkbhowever that seems mostly bundled up in a single patchset 15/22GB on ps1319:23
clarkbso maybe this is a one off19:23
fungihigh volume of changes/rechecks, or large logs for a single job?19:23
clarkbfungi: not sure yet, du still slow19:23
clarkbwill dig in though19:23
openstackgerritMerged openstack-infra/puppet-nodepool master: Create nodepool::launcher  https://review.openstack.org/43032919:23
fungiyeah, i have a feeling the current deletion work is slowing down your du, and vice versa19:23
clarkbshould I hold off while deletion goes?19:24
clarkbas that is what makes current jobs happy?19:24
fungimaybe, though there's also the possibility you'll find something we can delete directly to free up more space faster19:25
openstackgerritMerged openstack-infra/puppet-openstackci master: Add support for nodepool-launcher  https://review.openstack.org/43033919:25
clarkblike patchset 13 of change 404435 :)19:25
*** sdake has quit IRC19:26
sdagueok, anyone understand this pbr issue?19:26
sdaguehttp://paste.openstack.org/show/599102/19:26
sdaguethis is a 'pip3 install devstack-tools' on one of my devstack machines19:27
*** sdake has joined #openstack-infra19:27
clarkbfungi: pabelanger appears to be related to kolla's experiemntal jobs19:28
clarkbhttp://paste.openstack.org/show/599103/ probably a good idea to get them to reign that in if possible19:28
pabelangerI wonder if they are also copying images19:29
sdagueclarkb: also, we need https://review.openstack.org/#/c/433093 for ironic grenade jobs to work again, because they need a feature that's broken in the current release19:29
pabelangeryup19:30
pabelanger-rw-r--r-- 1 jenkins jenkins 3.1G Dec 14 09:40 centos-binary-registry.tar.gz19:30
pabelangerthat is the tarball they are coping to tarball.o.o19:30
*** sdake_ has quit IRC19:30
pabelangerlet me check there jobs and see what is going on19:30
fungifun, so they're also saving a copies of it to the logserver over and over?19:31
clarkbfungi: apeprs so19:31
*** rbrndt has quit IRC19:31
fungiwe can probably do a surgical find to delete all those and free up a bunch19:31
*** e0ne has joined #openstack-infra19:33
clarkbsdague: the problem appears to be that devstack-tools installs as lib/devstack but your version info is at lib/devstack_tools-version.dist-info19:33
*** e0ne has quit IRC19:33
clarkbsdague: so they don't line up, you may need to edit the pcakge name?19:33
sdagueclarkb: hmmm... this used to work19:33
clarkbsdague: I installed ina local venv then ls'd lib/python3.5/site-packages to see that19:34
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Enable mlock_tracker in gate  https://review.openstack.org/43447419:34
sdagueclarkb: so I need to redo the directory structure?19:35
clarkbsdague: I think that would be one way to fix it yes19:36
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Enable mlock_tracker in gate  https://review.openstack.org/43447419:36
clarkbsdague: should be easy enough to test if you mv then pip install ./repo/dir19:36
sdagueclarkb: so I'm super confused, because this totally was working, and is even used in the gate19:36
*** chlong has joined #openstack-infra19:36
sdaguemostly confused as to what changed to make it stop working19:37
*** john-davidge has quit IRC19:37
clarkbI don't know that pbr has changed19:37
clarkbmaybe setuptools changed?19:37
sdaguemaybe?19:37
clarkbther ewas a thread about releasing pbr and dhellmann said wait for ocata first, but maybe someone jumped the gun?19:38
sdaguepbr didn't seem to be released recently19:38
sdaguebut setuptools did 2 days ago19:38
*** EricGonczer_ has joined #openstack-infra19:38
mordred\o/19:38
fungiworth testing with an older setuptools to see if that was where the behavior change crept in19:39
fungithe changelog implies there were changes to finding/reading dist-info metadata19:40
*** e0ne has joined #openstack-infra19:40
mordredI'm not sure how this worked - this behavior (needing to match what's in __init__.py with what the name is in setup.cfg is not new at al19:42
mordredand has tripped up people before19:42
mordredI mean, that said - clearly it worked and now doesn't19:42
mordredbut it's a common mistake19:42
sdagueug19:42
sdaguestill didn't fix it19:43
sdagueoh, maybe I missed one reference19:43
mordredsdague: https://review.openstack.org/43447819:43
*** esikachev has joined #openstack-infra19:44
*** hashar has joined #openstack-infra19:44
mordredmaybe it worked before because something else installed a python package as "devstack" and that was what was being reported in __init__.py but nobody noticed they were off19:44
mordredand now nothing is instlaling anything as "devstack" so there is no entry in the pkg_resources system19:45
sdaguemordred: ah....19:45
sdaguemordred: interesting19:45
mordredsdague: did devstack itself used to install a thing and recently stopped?19:45
sdaguemordred: nope19:45
mordredhr19:45
mordredhrm19:45
sdaguethis is that thing, for the first time19:45
mordredwell - no clue how it ever worked but this bug has been latent in this repo since commit 119:46
sdaguemordred: yep, weird though, because this totally is being run in the grenade job and working19:46
sdagueanyway, thank you for the fix19:46
sdaguewell get that out shortly19:47
clarkbsdague: with python2 right? I wonder if its a 2 vs 3 weirdness somehow19:47
sdagueclarkb: I was pretty sure we were only doing python319:47
mordredit's not 2 vs 319:47
sdaguebut, maybe19:47
gordcmtreinish: just to verify, https://review.openstack.org/#/c/338435 is only being applied to master, ocata, newton?19:48
gordcit broke our gate because we look for uec suffix. just need to know which branches i need to backport to19:48
fungiclarkb: pabelanger: were one of you already in the process of syncing up with kolla liaisons?19:48
mtreinishgordc: and mitaka, https://review.openstack.org/36704919:48
fungii'm trying to get a count right now of how many centos-binary-registry.tar.gz copies we have on the logs volume to see what deleting them will free up19:49
clarkbnope happnes with python2 for me as well19:49
pabelangerfungi: clarkb: actually, I don't think it is an issue any more.  The experimental job has moved to post, but I no longer see us coping the tarball for the registry19:49
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate master: WIP/DNM: Testing grenade with Ironic multi-node job  https://review.openstack.org/43177719:49
clarkbpabelanger: ok so possibly a one off?19:49
gordcmtreinish: ack. we're backporting because of grenade?19:49
pabelangerclarkb: fungi http://logs.openstack.org/40/40130b2e7a591931ddac366dc6f8d085e1f83e37/post/kolla-dsvm-build-ubuntu-source-ubuntu-xenial/997bce3/logs/ is the job in question19:49
clarkbpabelanger: fungi though depending on how often they run the experiemnental and post jobs may still be an issue?19:49
fungipabelanger: okay, so odds are if we delete these they won't care (however there may not be enough of them to matter)19:49
mtreinishgordc: actually because we want to remove the uec support from tempest (it's been deprecated for 2 or 3 cycles)19:49
openstackgerritMonty Taylor proposed openstack-infra/shade master: Fix several concurrent shade gate issues  https://review.openstack.org/43442819:49
openstackgerritMonty Taylor proposed openstack-infra/shade master: Actually normalize nova usage data  https://review.openstack.org/43448119:49
openstackgerritMonty Taylor proposed openstack-infra/shade master: DNM - Patch to verify 2.0 usage payload behavior  https://review.openstack.org/43448219:49
*** tonytan4ever has quit IRC19:49
pabelangerclarkb: I think it was an issue when we did testing in experimental, how it moved to post around Jan 319:50
mtreinishgordc: https://review.openstack.org/#/c/338377/19:50
pabelangerso the job doesn't exist any more19:50
*** tonytan4ever has joined #openstack-infra19:50
gordcmtreinish: kk, thanks for background. will try to match on something more generic in our gate19:50
clarkbpabelanger: right just thinking if they ran the experimental jobs say 30 times thats roughly half a terabyte on its own19:51
*** chlong has quit IRC19:51
clarkbso may still be the problem19:51
pabelangerclarkb: yes, that is possible19:52
pabelangerhttps://review.openstack.org/#/c/404435/ would be the review to focus on19:52
pabelangerlet me see if I can find it19:52
clarkbpabelanger: well thats just one instance of it19:52
clarkband resulted in 15GB of logs on patchset 1319:52
clarkbso we'd need to look for other instances19:52
openstackgerritIan Cordasco proposed openstack-infra/project-config master: Add functional testing to Craton's check and gate  https://review.openstack.org/43439919:53
pabelangerclarkb: understood, 404435 was the main patch for testing experimental IIRC19:53
clarkbpabelanger: oh, except they only had massive logs the one time19:53
clarkband it wasn't in the last patchset either19:53
pabelangerright, I think something changed to stop the copy, but haven't figured out what19:54
jeblairclarkb, fungi, mordred: in addition to logging the size of things we copy to logs.o.o in zuulv3, we could probably *limit* the size -- ie, copy from worker to launcher; du; then bail if it is too large.  if we think we're likely to go that way in zuulv3, we could probably add that as a feature to v2.5 if we thought it necessary.19:54
mordredgordc: that broke the shade gate too - for the exact same reason19:56
clarkbpabelanger: gotcha19:56
clarkbI'm running another du scan to see if anything else like that pops up19:57
clarkband will probably continue to run them for a bit19:57
clarkbsince I can't really take a full world view look in reasonable amount of time19:57
*** mdurrant_ is now known as mdurrant19:57
gordcmordred: i used GLANCE_IMAGE_NAME=$(openstack image list | awk '/ cirros.* /{print $4; exit}') as workaround. not sure if that's the same issue you had.19:58
*** tonytan4ever has quit IRC19:58
mordredgordc: yah - similar - you may have problems with that on old clouds (because of ami/aki/ari existing - which is, I'm sure, why you were looking for the -uec string like we were) - but if the image change is backported everywhere, then I think your thing is golden19:59
*** tonytan4ever has joined #openstack-infra19:59
mordred(our lookup is in python, so I just look for startswtih('cirros') after I fail finding something that starts with cirros and ends with uec)20:00
openstackgerritIhar Hrachyshka proposed openstack-infra/puppet-elastic_recheck master: Enable recheck irc bot for neutron  https://review.openstack.org/43382820:00
*** Gorian- is now known as Gorian20:00
mtreinishihrachys_: ^^^ so I was thinking about that a bit, ideally what would you like e-r to report in channel?20:01
gordcmordred: hope so... else i need to google more bash stuff.20:01
mtreinishihrachys_: because the bot configuration might not be sufficient20:02
*** dharinic|lunch is now known as dharinic20:02
fungijeblair: that sounds like an excellent option for zuul20:02
mtreinishas in I might need to improve how we configure it in e-r20:02
pabelangerclarkb: Hmm, tripleo is copying etc/alternatives20:03
pabelangerwhich is about 911MB20:03
clarkbwat20:04
mordredmaybe they shoudl copy the sylinks and not the destinations20:04
clarkbpabelanger: for every host on every job?20:04
mordredare they using tar to copy?20:04
pabelangerclarkb: not sure, just checking20:04
pabelanger/srv/static/logs/29/431029/20:04
pabelangerwas most recent20:04
fungiwell, /etc/alternatives is symlinks to stuff in /usr/bin generally, so presumably the copying is dereferencing links as it goes20:04
*** esikachev has quit IRC20:04
mordredI could see copying /etc/alternatives to see what the symlinks point to - but copying the dereferneced links == bad20:04
*** erikmwilson has quit IRC20:04
pabelangerit is some java things20:05
mordredpabelanger: that'll be a nice thing to clean up20:05
mordredyup20:05
mordredjava alternatives on nodes20:05
mordredwill eat some space20:05
fungiouch, definitely20:05
mordredmaybe we shoudl "alias cp 'cp -d'" :)20:06
*** erikmwilson has joined #openstack-infra20:06
clarkbpabelanger: have a full path? I am not finding it through logs.o.o web browsing20:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Introducing Omni  https://review.openstack.org/43336320:07
mordredclarkb: isn't the full path /srv/static/logs/29/431029/ ?20:07
clarkbmordred: no there is not etc/alternatives there20:07
clarkbcheck/ and check-tripleo/ then the individual jobs20:07
*** Sukhdev has joined #openstack-infra20:08
mordrednod20:08
pabelanger/srv/static/logs/29/431029/26/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/3ca8868/logs/undercloud/etc/alternatives20:08
mordredclarkb: /srv/static/logs/29/431029/24/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ocata/c6090c9/logs/undercloud/etc/alternatives20:08
clarkbI guess I could switch to sshing onto static and use find but I like using the user accessible version as much as possible20:08
clarkbtyty20:08
mordredyah - things like /srv/static/logs/29/431029/24/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ocata/c6090c9/logs/undercloud/etc/alternatives/jre_1.7.0/lib/rhino.jar.gz are unhappy in logs :)20:08
mordredpabelanger: nice find20:08
pabelangerThe issue is, tripleo-ci jobs will grab everything, then exclude things20:08
pabelangerit should be the inverted, exclude everything, copy what is needed20:09
mordredpabelanger: well, we should at the very least add an exclude for /etc/alternatives20:09
pabelangeryes20:09
pabelangerwill patch20:09
mordredwhile we can get it swapped to what you said :)20:09
pabelangerhttps://review.openstack.org/#/c/434467/20:09
pabelangeris for /var/log/journal20:09
*** jkilpatr_ has joined #openstack-infra20:09
mordred++20:09
clarkbmy suggestion would be to specifically whitelist what you need out of /etc20:09
clarkbthis is what devstack-gate does and works well20:10
*** tonytan4ever has quit IRC20:10
fungior if you feel strongly that /etc should only contain small files, then don't follow symlinks you find in it20:10
*** tonytan4ever has joined #openstack-infra20:11
clarkbalso this is why I keep thinking it would be nice to have an /etc/openstack but there are still some projects that that wouldn't work with aiui20:11
*** Goneri has joined #openstack-infra20:11
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate master: Use new ara generate html command  https://review.openstack.org/43449220:11
pabelangerclarkb: yes!20:11
mordreddmsimard: ^^ that's accurate right?20:11
clarkbso /etc/openstack/nova /etc/openstack/neutron et al then just tar /etc/openstack and call it a day20:11
* dmsimard looks20:11
clarkbpabelanger: ya I tried to do this a couple years ago and discovered that like half our projects had hardcoded config path requirements20:12
clarkbpabelanger: and promptly had a sad and did something else more productive than fixing that20:12
dmsimardmordred: I had submitted https://review.openstack.org/#/c/433207/ already20:12
*** Sukhdev has quit IRC20:12
mordredoh!20:12
mordredI will abandon mine20:12
mordredclarkb: ^^ wanna +A dmsimard 's patch?20:13
*** jkilpatr has quit IRC20:13
* clarkb looks20:14
*** ldnunes has quit IRC20:17
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci master: Switch logs for TARCMD to include only specific logs  https://review.openstack.org/43449620:17
*** esikachev has joined #openstack-infra20:17
dmsimardmordred: when I deprecated it, I searched for usage in openstack land with codesearch.openstack.org and submitted patches everywhere20:17
dmsimard<3 codesearch.openstack.org20:17
mordredpabelanger: find /srv/static/logs -wholename '*etc/alternatives*' -exec 'rm -rf {}' would be fun :)20:17
mordreddmsimard: yay!20:17
openstackgerritIhar Hrachyshka proposed openstack-infra/puppet-elastic_recheck master: Enable recheck irc bot for neutron  https://review.openstack.org/43382820:17
mordreddmsimard: there are times when I really want an API to that so that I use it like find in a script :)20:18
dmsimardyeah, like "openstackgrep" or something20:18
openstackgerritMerged openstack-infra/project-config master: combine vitrage dsvm jobs as one pattern  https://review.openstack.org/43299020:18
mordreddmsimard: I mean - I'm 100% certain there is an API we could call20:19
mordreddmsimard: the website is all client-side javascript calling a go backend20:19
mordrediirc20:19
fungimordred: pabelanger: we might want to reduce the number of recursive scans we're performing on that filesystem. right now in have one deleting files older than 45 days and another that's been running for some 30 minutes now just trying to `find /srv/static/logs/ -name centos-binary-registry.tar.gz` which has system load up around 8 now20:20
mordreddmsimard: https://github.com/etsy/hound/blob/master/api/api.go ... /api/v1/search seems to be a thing20:20
mordredfungi: I will step away from logs.o.o20:21
dmsimardmordred: now you just need a client :)20:21
mordreddmsimard: I feel like we're each trying to nerd-snipe the other into being the one to write it :)20:21
*** esikachev has quit IRC20:21
*** rbrndt has joined #openstack-infra20:22
fungias it stands, the one find removing files >45 days still isn't able to delete data as fast as jobs are writing it20:22
*** salv-orlando has joined #openstack-infra20:22
fungiso i'm going to kill the other one i have trying to locate kolla's registry tarballs20:23
fungiin hopes that may speed things up a bit20:23
fungilooks like that's bringing system load back down a bit now20:25
fungiand the 45-day expiration find seems to maybe be picking up steam again, hard to be sure20:26
*** salv-orlando has quit IRC20:26
*** nadya has quit IRC20:26
*** esikachev has joined #openstack-infra20:27
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh  https://review.openstack.org/43102920:30
mordreddmsimard: import requests ; s=requests.get('http://codesearch.openstack.org/api/v1/search', params=dict(q='ara generate', stats='fosho', i='nope', repos='*', rng=':20', files=None)).json()['Results']20:30
mordreddmsimard: please enjoy that true and false are "fosho" and "nope" :(20:31
dmsimardlol20:31
clarkbhttps://review.openstack.org/#/c/410835/ was another change that had quite a bit of disk use. ~5GB20:34
clarkbbut appears to be lots of small cuts there and oslo.messaging just runs a ton of jobs20:34
*** rbuzatu has quit IRC20:35
*** ankur-gupta-f4 has left #openstack-infra20:36
*** esikachev has quit IRC20:36
*** jamielennox|away is now known as jamielennox20:36
sigmavirusmordred: why use dict(...)?20:37
fungiload average is down to ~2 now, but unfortunately the 45-day expiration find is still not deleting files fast enough to keep pace with uploads20:37
sigmavirusWhy do you hate dictionary literals?!20:37
sigmavirus=P20:37
*** ihrachys_ has quit IRC20:37
*** Sukhdev has joined #openstack-infra20:38
*** john-davidge has joined #openstack-infra20:38
fungiand load's spiked back up to ~5 again, so i guess it's not steady20:38
*** ihrachys has joined #openstack-infra20:39
*** rbuzatu has joined #openstack-infra20:39
mordredsigmavirus: too much typing :)20:39
openstackgerritIan Wienand proposed openstack-dev/pbr master: Remove testr entry point  https://review.openstack.org/42237920:40
mordredsigmavirus: but mostly just to troll you I think20:40
openstackgerritIan Wienand proposed openstack-dev/pbr master: Also generate XML coverage report  https://review.openstack.org/43407120:40
fungiyeah, dict() lets you skip some quoting ;)20:40
*** sdake has quit IRC20:41
fungiif you have >3 keys, it's fewer characters overall20:41
*** e0ne has quit IRC20:41
pabelangerfungi: clarkb: mordred: https://review.openstack.org/#/c/431029/ is the problem patch right now20:42
*** esikachev has joined #openstack-infra20:42
pabelangerit is what keeps burning through our HDD in logs.o.o20:42
*** john-davidge has quit IRC20:43
*** aviau has quit IRC20:43
*** rbuzatu has quit IRC20:43
*** aviau has joined #openstack-infra20:43
fungidoes it generate an inordinate amount of additional logging, or is it just because it has 27 patchsets over the course of a week?20:44
*** abompard has quit IRC20:44
fungianyway, to stop the bleeding, i could see deleting logs for all patchsets prior to the most recent one20:44
clarkblooks like ps24 was worth 18GB though the otehrs are much smaller20:44
clarkbheh check 21MB, check-tripleo 18GB20:45
fungithat may get us dar enough ahead of the curve that the expiration work will keep us above water until job volume trails off for the day20:45
fungis/dar/far/20:45
pabelangeryes20:46
smcginnisAre the POST_FAILURES I'm seeing from a full driver or something?20:46
pabelangerhttp://logs.openstack.org/29/431029/26/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/3ca8868/logs/undercloud/etc/20:46
pabelangertripleo is now copying the full contents of /etc from their undercloud20:46
clarkbit is interesting that if I du on check-tripleo it says 18GB20:46
clarkbif I du on the individual jobs under there its in megabytes20:46
clarkbso there is something funny going on in disk usage too I think20:47
mordredsmcginnis: yah - we have some jobs uploading giant amounts of data and we can't delete things quickly enough to keep up20:47
openstackgerritMerged openstack-infra/elastic-recheck master: Add query for bug 1664793  https://review.openstack.org/43443620:47
openstackbug 1664793 in OpenStack Compute (nova) "test_stamp_pattern timing out waiting for attached device to show up in guest" [Undecided,Confirmed] https://launchpad.net/bugs/166479320:47
*** r1chardj0n3s has joined #openstack-infra20:47
smcginnismordred: Joy!20:47
*** sdake has joined #openstack-infra20:47
*** sdake has quit IRC20:47
smcginnisWe're good at DOS'ing ourselves.20:47
clarkboh wait I am derp nevermind20:47
clarkbcd'd into the wrong dir20:47
mordredsmcginnis: maybe if cinder had a magical "please delete the files we don't want but not the ones we do want and do it in violation of the speed of light" feature?20:48
mordredsmcginnis: being well aware that cinder doesn't do filesystems of course ...20:48
smcginnismordred: I'll get a spec started for that right away! :)20:48
mordredsmcginnis: thanks!20:48
r1chardj0n3shey folks, any idea what's goimg on with https://review.openstack.org/#/c/427125 and the POST_FAILUREs of doom?20:48
AJaegersmcginnis: please include intelligent compression and removal of not needed files in the spec ;)20:48
smcginnisr1chardj0n3s: Just asked too. Log volumes are getting full.20:49
*** vsaienko has quit IRC20:49
jeblairfungi: maybe time for a #status notice?20:49
r1chardj0n3ssmcginnis: ah ok, thanks20:49
smcginnisjeblair: +120:49
fungir1chardj0n3s: we ran out of space to save logs from jobs and are in the process of blowing away anything older than 6 weeks to clear out enough space for current jobs20:49
r1chardj0n3sfungi: thanks20:49
smcginnisAJaeger: Maybe we can have it rewrite the blocks to change your data to what you actually want too. ;)20:49
*** ssbarnea has quit IRC20:49
fungir1chardj0n3s: though the volume is so high that we're not currently deleting as fast as new logs are uploaded. looking for other ways to pick up steam there20:49
*** sdake has joined #openstack-infra20:50
*** vsaienko has joined #openstack-infra20:50
mordredfungi: maybe a single recursive find looking for and deleting entire etc/alternatives dirs or kolla registry tarballs would get us larger chunks of files deleted at once?20:50
pabelangerclarkb: ya, 18GB for all jobs, about 4GB for each undercloud20:50
*** esikachev has quit IRC20:51
pabelangerclarkb: from what I understand, it is a recent change for to help tripleo debug failures20:51
clarkbpabelanger: unfortuniately it is now causing failures :P20:51
mordredwhich would explain why we're out of space all of a sudden20:51
pabelangerindeed20:52
*** abompard has joined #openstack-infra20:52
*** abompard has joined #openstack-infra20:52
fungistatus notice We're currently battling an increase in log volume which isn't leaving sufficient space for new jobs to upload logs and results in POST_FAILURE in those cases; recheck if necessary but keep spurious rebasing and rechecking to a minimum until we're in the clear.20:52
fungisomething like that? ^20:52
pabelangeryes20:52
jeblairfungi: lgtm20:52
*** zara_the_lemur__ has left #openstack-infra20:53
pabelangerwe could maybe set max server to 0 for tripleo-test-cloud-rh1, until we get a large chunk of data restored20:53
mordredpabelanger: I thought they were running a bunch of things in normal gate now?20:53
mordredfungi: ++20:53
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: tripleo-ui: add openstack-server-release-jobs  https://review.openstack.org/43451020:54
clarkbpabelanger: we are and those are in few MB range of log data20:54
clarkber mordred ^20:54
pabelangermordred: they are, but this looks limited to ovb jobs, which run on tripleo-test-cloud-rh120:54
mordredah20:54
mordredyah - I think dropping max server to 0 for a sec while we clean things should help us not fight new uploads maybe?20:54
jeblairyeah, we limit log uploads from tox runs to 50MB.  that's kind of what we considered a 'large amount of data' at one point.20:54
mordredjeblair: nobody would ever need more than 64K of ram you know20:55
fungi#status notice We're currently battling an increase in log volume which isn't leaving sufficient space for new jobs to upload logs and results in POST_FAILURE in those cases; recheck if necessary but keep spurious rebasing and rechecking to a minimum until we're in the clear.20:55
openstackstatusfungi: sending notice20:55
mordredthat's 64 THOUSAND bytes!!!20:55
pabelangermaybe time to update rsync to limit data too?20:55
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: tripleo-ui: add openstack-server-release-jobs  https://review.openstack.org/43451020:55
jeblairmordred: that's more than 400 tweets20:55
*** sdake has quit IRC20:56
-openstackstatus- NOTICE: We're currently battling an increase in log volume which isn't leaving sufficient space for new jobs to upload logs and results in POST_FAILURE in those cases; recheck if necessary but keep spurious rebasing and rechecking to a minimum until we're in the clear.20:56
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Enable peakmem_tracker in gate  https://review.openstack.org/43451120:57
*** saibarspeis has quit IRC20:57
*** tnovacik has quit IRC20:57
EmilienMpabelanger: sorry for the problem about logs :-/ we'll make our possible to reduce the number of logs we're saving and are open to changes20:57
*** rfolco has quit IRC20:57
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Disable tripleo-test-cloud-rh1 cloud  https://review.openstack.org/43451220:58
pabelangermordred: fungi: ^ the extreme option20:58
openstackstatusfungi: finished sending notice20:58
pabelangerEmilienM: it appears limited to OVB jobs20:59
EmilienMpabelanger: wait, what is the impact of that?20:59
EmilienMpabelanger: can we have a quick workaround in tripleo-ci in the meantime?20:59
*** vsaienko has quit IRC20:59
*** adriant has joined #openstack-infra20:59
EmilienMslagle, bnemec ^ fyi20:59
pabelangerEmilienM: nothing decided yet, but it would stop OVB jobs from running21:00
pabelangeruntil we can reclaim HDD space21:00
EmilienMok, let's try to find a short term solution for now21:00
mordredEmilienM: currently we're producing about 4G of logs (and other files) for each undercloud21:01
EmilienMI know I know about the number21:01
EmilienMI know it's terrible etc21:01
mordredkk. sorry - just trying to tl;dr if it was busy scrollback21:01
*** mdorman has left #openstack-infra21:01
jeblairwe could land that and revert it quickly as soon as we have something better.21:02
fungiEmilienM: a major cause is apparently that all of /etc is being copied, and symlinks (including stuff in /etc/alternatives) are dereferenced when doing so resulting in copies of all sorts of stuff from /usr you really don't need21:02
EmilienMpabelanger: ok so https://review.openstack.org/#/c/434467 is ready to land once we have the OVB jobs results21:03
*** erikmwilson has quit IRC21:04
openstackgerritMorgan Fainberg proposed openstack-infra/shade master: Convert use of .register_uri to .register_uris  https://review.openstack.org/43451621:04
EmilienMpabelanger: if we exclude /etc/alternatives for now, is it a good start?21:07
*** erikmwilson has joined #openstack-infra21:08
pabelangerEmilienM: we need to figure out what changed to start uploading: http://logs.openstack.org/29/431029/24/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ocata/c6090c9/logs/undercloud/etc/21:09
*** mhickey has joined #openstack-infra21:09
clarkbEmilienM: yse, however my suggestion earlier was you need to whitelist what you want out of /etc21:09
pabelangerbecause that is 4GB of data each patch21:09
mordredEmilienM: yah. I think that and also the /var/log/journal patch21:09
clarkbdon't copy everything just because21:09
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck master: Add support to irc bot for filtering messages by repo  https://review.openstack.org/43451921:09
mordredI think just blacklisting those two while you work up the correct whitelist21:09
mordredwill be very helpful21:09
*** jamesdenton has quit IRC21:09
mtreinishihrachys: ^^^ I think that'll do it21:09
mtreinishihrachys: heh, I also just noticed you're not in openstack-qa :)21:10
EmilienMclarkb: I agree, but it might take some time to do it21:10
clarkbI mean21:10
EmilienMmordred: ok21:10
clarkbthere is already a whitelist in devstack/devstack-gate21:10
clarkbjust reuse that?21:10
EmilienMmordred: on it21:10
EmilienMclarkb: we'll consider this option seriously21:10
ihrachysmtreinish: should I? :) lemme add it to my list.21:10
EmilienMclarkb: let me grab some people and estinguish this fire. We'll fix the root cause later, this week21:11
*** raildo has quit IRC21:11
pabelangerI'm going to start purging http://logs.openstack.org/29/431029 for some HDD space21:11
fungithanks pabelanger! that's already making a significant difference21:12
fungi17gib free and climbing21:12
openstackgerritKaitlin Farr proposed openstack-infra/project-config master: Enable signature verification for barbican-tempest-plugin gate  https://review.openstack.org/43438421:13
*** ijw has joined #openstack-infra21:14
pabelangerfungi: ya21:14
jeblairi approved the disable tripleo-test-cloud-rh1 disable; if we want to stop that, we should act fast.  :)21:16
*** clayg_ is now known as clayg21:16
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck master: Add support to irc bot for filtering messages by repo  https://review.openstack.org/43451921:16
fungifingers crossed that this surge in available space will carry us through until the 45-day expiration pass is able to scan further through the filesystem21:17
*** Apoorva has quit IRC21:17
pabelangerokay, deleted 43102921:17
pabelangergot us 24GB21:18
clarkbdid we also manage to delete the 15GB of kolla images?21:18
pabelangerI didn't, but can check again21:18
clarkbeven if we just delete that one set of experimental job logs it will free up ~15GB so may be worth it for headroom21:19
fungiclarkb: logs from a single tripleo change21:19
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci master: Exclude etc/alternatives  https://review.openstack.org/43452321:19
pabelangerk, doing kolla logs now21:19
fungibut yes, we should do a separate find to remove those kolla repository tarballs from their experimental job logs21:20
fungii gave up after 30 minutes of waiting for it to try to find them because i didn't want to slow down the 45-day expiration pass even more21:20
*** ijw has quit IRC21:20
openstackgerritMerged openstack-infra/project-config master: Disable tripleo-test-cloud-rh1 cloud  https://review.openstack.org/43451221:22
*** vsaienko has joined #openstack-infra21:22
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck master: Add support to irc bot for filtering messages by repo  https://review.openstack.org/43451921:23
openstackgerritJeremy Stanley proposed openstack-infra/puppet-openstackci master: Reduce job log retention to 45 days  https://review.openstack.org/43452421:23
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Revert "Disable tripleo-test-cloud-rh1 cloud"  https://review.openstack.org/43452521:23
pabelangerwhen we are ready^21:23
openstackgerritAntoine Musso proposed openstack-infra/zuul master: Expose commitMessage as a Change attribute  https://review.openstack.org/22279121:23
*** ijw has joined #openstack-infra21:24
*** annegentle has quit IRC21:24
*** gildub has joined #openstack-infra21:25
*** kgiusti has quit IRC21:26
*** askb has joined #openstack-infra21:27
clarkbfungi: ya was just thinking doing the specific one we know about would be useful21:28
pabelangerup to 45GB now21:28
*** Apoorva has joined #openstack-infra21:29
*** esikachev has joined #openstack-infra21:29
fungiclarkb: oh, yep sure21:31
EmilienMtbh, I don't know what to do now. It seems like the /etc/alternatives thing was done in a single patch: https://review.openstack.org/#/c/431029/21:32
EmilienMslagle is making sure we don't have this dir in multinode jobs but we think it won't do anything since we didn't have the dir before21:33
EmilienMhttps://review.openstack.org/#/c/434523/21:33
EmilienMand pabelanger is exclusing the journal file: https://review.openstack.org/#/c/434467/21:33
EmilienMwhat else we can do on short term?21:33
EmilienMexcept blocking this patch: https://review.openstack.org/#/c/431029/21:33
*** Jeffrey4l has quit IRC21:34
EmilienMAJaeger: can you look https://review.openstack.org/#/c/434510 please? I replied, I need your guidance21:34
*** Jeffrey4l has joined #openstack-infra21:35
clarkbEmilienM: I think once fungi's expiration pass is complete we will be in a better place to audit all the things more quickly and accurately21:35
fungiEmilienM: all that sounds great to me21:35
*** kzaitsev_mb has joined #openstack-infra21:35
openstackgerritSean Dague proposed openstack-infra/devstack-gate master: Upgrade devstack-tools to 0.2.0  https://review.openstack.org/43309321:35
clarkbthen hopefulyl we will have more feedback for tripleo and others21:35
openstackgerritSean Dague proposed openstack-infra/devstack-gate master: Upgrade devstack-tools to 0.2.1  https://review.openstack.org/43309321:35
*** jkilpatr_ has quit IRC21:35
EmilienMclarkb: ok21:36
openstackgerritMerged openstack-infra/devstack-gate master: Replace "ara generate" by "ara generate html"  https://review.openstack.org/43320721:36
fungiwe're past 0.3% free now!21:38
*** burgerk has quit IRC21:38
*** rbuzatu has joined #openstack-infra21:40
*** esikachev has quit IRC21:41
openstackgerritMerged openstack-infra/system-config master: Add nl01.openstack.org server  https://review.openstack.org/43032421:41
*** dimtruck is now known as zz_dimtruck21:42
EmilienMdo we have alerts (email or something) that says when we reach more than 80%, etc?21:43
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Run zuul v3 launcher  https://review.openstack.org/43334721:44
fungiEmilienM: no, because we're nearly always there21:45
*** sdake has joined #openstack-infra21:46
fungiour main recourse is continually shrinking log retention21:46
EmilienMfungi: ok, got it21:46
pabelangerEmilienM: http://logs.openstack.org/00/420100/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha-oooq-nv/b3365b3/logs/undercloud/var/log/oooq/ is another entry, using a large amount of data21:46
pabelangerEmilienM: looks to be a lot of duplicate data21:46
EmilienMsshnaidm: ^21:46
fungiused to be 6 months, now we're only able to keep 6 weeks :/21:46
sshnaidmEmilienM, yeah, I found the problem21:46
sshnaidmEmilienM, it's related to one of patches merged to extras today21:47
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Inherit playbooks and modify job variance  https://review.openstack.org/43048321:47
pabelanger /srv/static/logs/00/420100 alone is 6.1GB over 6 patchsets21:47
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Split merger and launcher git roots  https://review.openstack.org/43045621:47
*** annegentle has joined #openstack-infra21:49
*** esikachev has joined #openstack-infra21:49
sdaguepabelanger: yeh, it looks like the whole filesystem is getting uploaded21:49
openstackgerritRussell Bryant proposed openstack-infra/project-config master: networking-ovn: Reorganize CI jobs.  https://review.openstack.org/39468421:49
sdagueon every run21:50
mordredrelated to our earlier conversation about putting an rsync limit in - I think we can safely say that 1G of log upload for a single patch is _definitely_ too much21:50
sdaguemordred: ++21:50
*** eharney_ has joined #openstack-infra21:50
*** eharney has quit IRC21:50
mordredI'm pretty sure the limit wants to be lower than that21:50
*** eharney_ is now known as eharney21:50
pabelangerinteresting, stackviz is about 20MB too21:51
pabelanger20M ./stackviz21:51
pabelangerthat is from gate-tempest-dsvm-neutron-full-ubuntu-xenial21:51
EmilienMpabelanger: giving you a bit of background here. The issue was not directly related to regular OVB & multinode jobs but to TripleO Quickstart transition. I apologize for that, but this tooling is still experimental. The "stable" tooling doesn't consume much storage for logs, unless someone proves it the opposite21:51
jeblairmordred: so, somewhere between 64k and 1gb, you're saying?21:51
openstackgerritMorgan Fainberg proposed openstack-infra/shade master: Convert test_object to use .register_uris  https://review.openstack.org/43453221:51
fungimordred: trying to think of how that should work to avoid hamstringing projects, maybe if there's to much then instead only upload a list of the files and their sizes (in a machine-readable format for easier investigation)?21:51
fungis/to much/too much/21:52
mordredfungi: ++21:52
jeblairfungi: that's a good idea.  and of course, always the console log.21:52
mordredfungi: I tihnk that's a great idea21:52
mordredjeblair: yah :)21:52
fungiright, always the console log21:52
EmilienMpabelanger: in other words, we'll work with TripleO QuickStart folks to improve log collection but the current OVB / multinode jobs that don't use quickstart seems to work fine and don't collect the world.21:52
EmilienMfungi, clarkb, mordred: I found useful to share that ^21:53
fungithanks EmilienM!21:53
*** esikachev has quit IRC21:54
mordredEmilienM: nod21:54
EmilienMI wouldn't blame TripleO CI as a whole, but I would just help folks working on Quickstart to get log collection better and probably re-use what has been done by devstack gate21:54
jheskethMorning21:54
fungilooks like we're no longer losing the battle for free space, so can probably turn the tripleo cloud back on whenever?21:54
EmilienMwe've found out that the main root cause was: https://review.openstack.org/#/c/431029/21:54
*** eroux has quit IRC21:54
*** esikachev has joined #openstack-infra21:55
pabelangerfungi: 434525 is the revert21:56
*** zz_dimtruck is now known as dimtruck21:56
pabelangerit's been about 20mins since tripleo-test-cloud-rh1 has been disabled21:56
*** dprince has quit IRC21:57
*** annegentle has quit IRC21:57
*** panda|bbl is now known as panda21:58
*** claudiub has joined #openstack-infra21:58
*** tonytan4ever has quit IRC21:59
*** esikachev has quit IRC21:59
openstackgerritKaitlin Farr proposed openstack-infra/project-config master: Enable signature verification for barbican-tempest-plugin gate  https://review.openstack.org/43438422:01
*** john-davidge has joined #openstack-infra22:01
*** claudiub|2 has quit IRC22:01
*** esikachev has joined #openstack-infra22:02
*** Krenair has quit IRC22:02
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Re-enable test_queue_names  https://review.openstack.org/42988322:03
*** ihrachys_ has joined #openstack-infra22:04
*** rwsu has quit IRC22:04
*** panda_ has joined #openstack-infra22:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove misleading log statements  https://review.openstack.org/43397022:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Squelch ref-replication gerrit warnings  https://review.openstack.org/43395922:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add Git driver  https://review.openstack.org/43394222:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove gearman settled check  https://review.openstack.org/43394322:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add (minimal) support for topic-changed event  https://review.openstack.org/43396622:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename zuul-server zuul-scheduler  https://review.openstack.org/43398022:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Correct getGitwebUrl  https://review.openstack.org/43396422:05
*** panda is now known as Guest4530322:05
*** ihrachys has quit IRC22:05
*** Guest45303 has quit IRC22:05
*** Goneri has quit IRC22:05
*** panda_ is now known as panda22:05
*** trown is now known as trown|outtypewww22:06
*** john-davidge has quit IRC22:06
clarkbok last chance to veto my ptg dinner location/time/seatcount selections22:06
*** esikachev has quit IRC22:06
clarkbI'm gonna grab a late lunch then give restaurant(s) a call22:06
*** esikachev has joined #openstack-infra22:07
*** vsaienko has quit IRC22:08
*** Goneri has joined #openstack-infra22:09
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh  https://review.openstack.org/43102922:09
*** ihrachys_ has quit IRC22:10
*** esikachev has quit IRC22:11
*** jkilpatr_ has joined #openstack-infra22:12
*** EricGonczer_ has quit IRC22:14
*** rbuzatu has quit IRC22:14
*** EricGonczer_ has joined #openstack-infra22:14
*** ihrachys has joined #openstack-infra22:15
*** thorst_ has quit IRC22:16
r1chardj0n3sthanks again for keeping the gates going, #infra!22:16
*** r1chardj0n3s has left #openstack-infra22:16
*** gouthamr has quit IRC22:22
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: replace toci_gate_test.sh  https://review.openstack.org/43102922:22
*** ihrachys has quit IRC22:23
*** salv-orlando has joined #openstack-infra22:23
*** EricGonczer_ has quit IRC22:24
*** esikachev has joined #openstack-infra22:26
*** salv-orlando has quit IRC22:28
*** esikachev has quit IRC22:30
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add Git driver  https://review.openstack.org/43394222:30
*** rbuzatu has joined #openstack-infra22:31
*** hashar has quit IRC22:32
*** rhallisey has quit IRC22:32
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove gearman settled check  https://review.openstack.org/43394322:32
*** masayukig is now known as masayukig_22:33
*** sshnaidm is now known as sshnaidm|off22:34
*** vsaienko has joined #openstack-infra22:35
*** rbuzatu has quit IRC22:36
*** Krenair has joined #openstack-infra22:36
*** matt-borland has quit IRC22:43
pandapabelanger: you there ?22:43
openstackgerritMonty Taylor proposed openstack-infra/shade master: Actually normalize nova usage data  https://review.openstack.org/43448122:44
mordredclarkb: when you get back from restaurants - could you do me a friendly and look at https://review.openstack.org/#/c/434428/ ? I've been trying to not bother people with shade patches - but this one is fixing gate breaks from today22:45
openstackgerritAndrea Frittoli proposed openstack-infra/elastic-recheck master: Add query for OOM issue on RAX 1664953  https://review.openstack.org/43423822:45
mordred(it's not a scary patch)22:45
*** esikachev has joined #openstack-infra22:46
clarkbmordred: I can, also discovering restaurants are hard22:46
clarkbmordred: first option says we need one check for group that big. So if you want to load up your corp card... :P22:46
clarkbcurrently trying to find phone number for other restaurant but its in a hotel so this is harder than it should be22:47
mordredclarkb: yah - I don't think it'll be a problem (what's our headcount?)22:47
clarkbmordred: 1422:47
mordredyah - happy shouldn't be a problem22:47
pabelangerpanda: some what22:48
pandafungi: thanks22:49
pandapabelanger: oh, well sorry for the mess, I just wanted to say you can wipe all the data relative to those patches22:49
openstackgerritMerged openstack-infra/project-config master: Revert "Disable tripleo-test-cloud-rh1 cloud"  https://review.openstack.org/43452522:49
*** eharney has quit IRC22:49
pabelangerpanda: np! Thanks for looking into the issue22:50
*** esikachev has quit IRC22:50
fungionce the 45-day expiration completes (which i anticipate being sometime tomorrow) i'll do a dev ml post with a redux and current state on retention vs quantity22:52
*** dave-mccowan has quit IRC22:53
*** gouthamr has joined #openstack-infra22:54
*** adrian_otto1 has joined #openstack-infra22:59
*** vsaienko has quit IRC22:59
clarkbfungi: it looks like we have steady stated around 42GB free23:00
fungiyep23:00
smcginnisIs logging still an issue, or is it safe to recheck some things?23:00
fungiought to grow wider as the ci load lessens for the day23:01
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Allow nl01.o.o to access zookeeper  https://review.openstack.org/43455023:01
*** rbuzatu has joined #openstack-infra23:01
*** adrian_otto has quit IRC23:01
pabelangerclarkb: fungi: jeblair: ^ firewall rule to allow nl01.o.o to talk to zookeeper23:01
pabelangeralso, nl01.o.o is online23:01
pabelangerjust waiting for puppet to run and confirm in cacti23:01
fungismcginnis: not out of the woods yet, but the errors should have subsided an hour or so ago23:01
*** salv-orlando has joined #openstack-infra23:02
jeblairpabelanger: lgtm23:04
smcginnisfungi: Great, thanks!23:05
*** rbuzatu has quit IRC23:05
*** esikachev has joined #openstack-infra23:07
*** _d34dh0r53_ is now known as d34dh0r5323:09
*** mdrabe has quit IRC23:10
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove misleading log statements  https://review.openstack.org/43397023:11
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Squelch ref-replication gerrit warnings  https://review.openstack.org/43395923:11
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add (minimal) support for topic-changed event  https://review.openstack.org/43396623:11
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename zuul-server zuul-scheduler  https://review.openstack.org/43398023:11
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Correct getGitwebUrl  https://review.openstack.org/43396423:11
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Temporarily disable test_dependent_behind_dequeue  https://review.openstack.org/43455323:11
clarkbmordred: shade fixes lgtm23:12
*** esikachev has quit IRC23:12
mordredclarkb: thanks!23:13
*** spzala has quit IRC23:17
*** adrian_otto1 has quit IRC23:17
*** thorst_ has joined #openstack-infra23:17
*** gyee has joined #openstack-infra23:18
pabelanger#status log nl01.openstack.org (nodepool-launcher) is now online. Nodepool services are disabled.23:18
openstackstatuspabelanger: finished logging23:18
pabelangerinfra-root:^23:18
*** adrian_otto has joined #openstack-infra23:19
*** mriedem has quit IRC23:19
*** tphummel has quit IRC23:20
*** ihrachys has joined #openstack-infra23:21
*** wolverin_ has quit IRC23:21
*** thorst_ has quit IRC23:21
*** wolverineav has joined #openstack-infra23:22
*** ihrachys has quit IRC23:22
*** rbuzatu has joined #openstack-infra23:23
*** baoli has quit IRC23:23
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Temporarily disable test_dependent_behind_dequeue  https://review.openstack.org/43455323:23
*** esikachev has joined #openstack-infra23:24
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Squelch ref-replication gerrit warnings  https://review.openstack.org/43395923:24
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Correct getGitwebUrl  https://review.openstack.org/43396423:25
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add (minimal) support for topic-changed event  https://review.openstack.org/43396623:25
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Re-enable test_dib_upload_fail test  https://review.openstack.org/43323523:26
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove misleading log statements  https://review.openstack.org/43397023:26
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Rename zuul-server zuul-scheduler  https://review.openstack.org/43398023:26
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Re-enable working test_builder.py tests  https://review.openstack.org/43326223:27
*** thorst_ has joined #openstack-infra23:27
*** rbuzatu has quit IRC23:27
*** wolverineav has quit IRC23:28
*** esikachev has quit IRC23:28
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Re-enable test_diskimage_build_only test  https://review.openstack.org/43326523:28
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Re-enable test_image_upload_fail test  https://review.openstack.org/43327023:28
*** masayukig_ is now known as masayukig23:28
*** panda is now known as panda|zZ23:29
*** gordc has quit IRC23:29
*** marst has quit IRC23:29
*** marst has joined #openstack-infra23:29
*** denisra_ has quit IRC23:30
*** Sukhdev has quit IRC23:31
*** thorst_ has quit IRC23:32
*** thorst_ has joined #openstack-infra23:32
pabelangerstill not seeing snmp requests to nl01.o.o23:33
pabelangernot sure why that is ATM23:33
*** marst has quit IRC23:34
clarkbdid it get added as a host in cacti?23:34
pabelangeryup23:34
pabelangerI can also see nl01.o.o in /var/lib/cacti/devices23:35
pabelangerguess a waiting game now23:35
*** dingyichen has joined #openstack-infra23:35
*** thorst_ has quit IRC23:37
*** masayukig is now known as masayukig_23:38
clarkbup to 47GB free23:38
pabelangerokay, working now23:38
clarkbI guess the tapering off must be happening23:38
pabelangerdidn't wait long enough23:39
pabelangerclarkb: I think tomorrow I'll see about adding more space to tarballs.o.o23:40
pabelangerhttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1055&rra_id=all23:40
pabelangerjust to be safe23:40
clarkbpabelanger: we can't23:40
pabelangerOh23:40
pabelangerwhy is that?23:40
clarkbwe are at our limit of attached devices and every volume is max size for individual volume23:41
mordredyup23:41
clarkbwe'd have to host tarballs on a different host then move the volume(s)23:41
pabelangerOIC23:41
pabelangerk, maybe we should figure that out at PTG23:41
mordredmechanisms by which we could grow the storage are to move to an AFS solutoin with multiple fileservers - or to run something liek ceph on top of multiple fileservers23:41
clarkbpabelanger: Xen has limit of 16 devices, 2 are taken up by base VM (one is / and one is swap iirc) and then we have 14 1TB cinder volumes with lvm over the top23:42
mordredeach of those increase the level of complexity, as we'd be putting complex network things over top of cloud complex network things23:42
*** marst has joined #openstack-infra23:42
clarkbI guess another possible fix is beg cloud provider to give us bigger volumes23:44
clarkbbut I think they are size restricted for raisins23:44
mordredif we did multiple fileservers behind an AFS cell - we'd still have to allocate storage to locatoins, and that might get tedious with our dir hashing structure23:44
clarkband not just arbitrarily done that way23:44
*** sdake has quit IRC23:45
*** tphummel has joined #openstack-infra23:45
mordredbut it would have the benefit that we already know how to publish to those - if we did ceph there are more moving parts for us to deal with, but it has the benefit of presenting as single contiguous region of storage that we don't have to split between a bunch of directories23:45
mordredboth seem like a non-zero amount of work23:46
clarkbmordred: do you know if ceph allows for online resize?23:46
mordredBUT - I agree with pabelanger - we should at least talk about it at the PTG23:46
clarkbseems like it should considering its block device is built on object storage23:46
mordredclarkb: I do not know - but I _think_ so23:47
*** salv-orlando has quit IRC23:47
*** esikachev has joined #openstack-infra23:47
jeblairyeah, i imagine if we used afs, we'd do something like make 100 volumes, and write a script to manage them, balancing/moving them as needed to other fileservers23:47
mordredjeblair: yup23:47
jeblair(afs is meant to scale to tens or hundreds of thousands of volumes, so that's still peanuts)23:48
mordredjeblair: you know - russ albery wrote that program in fortran to calculate afs volume rebalances23:48
jeblairmordred: i imagine it is correct.23:48
clarkbjeblair: just all within a millisecond of each other :)23:48
mordredjeblair: I cannot imagine it is incorrect23:48
openstackgerritMerged openstack-infra/zuul master: Fix missing mutex release when aborting builds  https://review.openstack.org/38498023:49
mordredclarkb: the nice thing about _this_ application is that it has a much more resilient consumption demands than our mirror volumes23:49
mordredand we won't make copies of files into caches until they're actually looked at, which tails off pretty rapidly23:50
clarkbya23:50
mordredbut - yeah - there are a bunch of pros and cons in a bunch of different directions to discuss23:50
jeblairi'll note that "talk about this in more detail" is pretty exactly the feedback jhesketh gave on https://review.openstack.org/26992823:50
clarkbin an ideal world I think I'd ask manilla for 50TB and then when that fills up consider spring cleaning and if that isn't feasible ask it to grow to 100TB23:51
*** esikachev has quit IRC23:51
mordredclarkb: I keep waiting for the day when we live in an ideal world :)23:51
mordredjeblair: and yah23:51
clarkbI need to record norah saying "Turtle" then when I think of turtles all the way down I can play that and laugh23:52
mordredthe nice thing is I think we're all in a better position to discuss that spec in more detail now23:52
mordredclarkb: hahaha23:52
clarkbshe like rolls her els and its turtlertlertlertletle when she says it23:53
*** spzala has joined #openstack-infra23:55
fungipabelanger: make sure to put tarballs.o.o discussion on our ptg etherpad if you haven't already23:55
pabelangerfungi: roger23:55
mordred++23:55
*** rbrndt has quit IRC23:57
*** mhickey has quit IRC23:58
*** spzala has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!