Saturday, 2016-09-10

*** ansiwen has quit IRC00:00
*** mdbooth has quit IRC00:00
*** Ravikiran_K has quit IRC00:00
raddaouiclarkb: fungi pabelanger, cloud8 is running again00:01
*** mdbooth has joined #openstack-infra00:01
*** gyee has quit IRC00:02
*** ansiwen has joined #openstack-infra00:02
raddaouiyall have a good weekend00:02
sdakeoh nm joh nm 3 seconds00:04
* sdake groans00:04
*** dteselkin has quit IRC00:06
*** tqtran has quit IRC00:09
*** mbacchi has joined #openstack-infra00:12
*** xarses has joined #openstack-infra00:12
*** dteselkin has joined #openstack-infra00:15
*** jkilpatr has quit IRC00:17
*** roxanagh_ has joined #openstack-infra00:19
*** fguillot has quit IRC00:22
*** roxanagh_ has quit IRC00:27
*** fguillot has joined #openstack-infra00:29
*** roxanagh_ has joined #openstack-infra00:29
fungimwhahaha: well, i'd be keen on not leaving this broken for most of infra's puppet modules all weekend (selfishly, as i have things i need to get merged for ongoing efforts of ours). i'll pin it for the weekend and we can revisit on monday. maybe the bundler team will have a fix identified and released by then00:30
*** kaisers1 has joined #openstack-infra00:38
*** kaisers has quit IRC00:40
*** tphummel has quit IRC00:42
*** gyee_ has quit IRC00:45
*** roxanagh_ has quit IRC00:46
openstackgerritMerged openstack-infra/system-config: Give gate-infra-puppet-apply-fedora-24 some work  https://review.openstack.org/36671800:49
*** lifeless_ has joined #openstack-infra00:50
*** lifeless has quit IRC00:50
*** fguillot has quit IRC00:51
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Pin bundler to 1.12.5 for some jobs  https://review.openstack.org/36826700:51
*** Sukhdev has quit IRC00:51
*** fguillot has joined #openstack-infra00:51
*** baoli has joined #openstack-infra00:52
*** IlyaG has joined #openstack-infra00:54
*** roxanagh_ has joined #openstack-infra01:05
*** rajinir has quit IRC01:05
*** armax has quit IRC01:07
*** IlyaG has quit IRC01:08
*** roxanagh_ has quit IRC01:09
*** fguillot has quit IRC01:16
pabelangerclarkb: I guess the next fix for nodepool-builder is not to upload a new image (with the same label), is an existing is already started.01:25
pabelangerclarkb: that's the issue we are having with ovh-bhs1 now01:25
clarkbpabelanger: ya but maybe also get uploads to there fixed01:25
pabelangeragreed01:26
pabelangerclarkb: what was the syntax you used to query gearman to see which images were uploading?01:26
pabelangerwe still have 54 images to upload01:27
pabelangerand figure something is blocked again01:27
*** zhurong has joined #openstack-infra01:29
*** ddieterly has joined #openstack-infra01:30
*** thorst has joined #openstack-infra01:31
*** Mary has joined #openstack-infra01:31
*** Mary is now known as Guest5288901:32
*** Guest52889 has quit IRC01:36
clarkbpabelanger: echo status | nc localhost 4730 | grep image-upload01:37
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: Put the regex first  https://review.openstack.org/36827401:37
*** askb has quit IRC01:39
*** zhurong has quit IRC01:42
*** Goneri has quit IRC01:43
*** AnarchyAo has joined #openstack-infra01:47
nibalizerfungi: good ol bundler trollin01:50
bkeroI wonder how many times we can blame bundler for breaking CI01:53
bkeroIt seems like lots.01:53
*** rfolco_ has joined #openstack-infra01:56
*** rfolco_ has quit IRC01:56
*** zhurong has joined #openstack-infra02:09
*** zz_dimtruck is now known as dimtruck02:12
stevemaro/02:16
*** zhurong has quit IRC02:16
*** armax has joined #openstack-infra02:24
*** salv-orl_ has joined #openstack-infra02:29
*** gouthamr has quit IRC02:31
*** salv-orlando has quit IRC02:32
*** ddieterly has quit IRC02:41
*** mbacchi has quit IRC02:44
*** thorst has quit IRC02:47
*** ddieterly has joined #openstack-infra02:50
*** mbacchi has joined #openstack-infra02:51
*** tonytan4ever has joined #openstack-infra02:55
*** tonytan_brb has quit IRC02:58
*** tqtran has joined #openstack-infra03:08
*** baoli_ has joined #openstack-infra03:08
*** edmondsw has quit IRC03:09
*** baoli has quit IRC03:11
*** tqtran has quit IRC03:12
*** rhallisey has quit IRC03:16
*** ddieterly has quit IRC03:18
*** mbacchi has quit IRC03:19
*** baoli_ has quit IRC03:19
*** woodster_ has quit IRC03:19
*** yamahata has quit IRC03:23
*** zhurong has joined #openstack-infra03:25
*** bexelbie has quit IRC03:28
*** ociuhandu has joined #openstack-infra03:31
*** thorst has joined #openstack-infra03:33
*** zhurong has quit IRC03:34
*** ociuhandu has quit IRC03:35
*** yamahata has joined #openstack-infra03:36
*** armax has quit IRC03:37
*** thorst has quit IRC03:41
*** ociuhandu has joined #openstack-infra03:46
*** zhurong has joined #openstack-infra03:46
*** yamahata has quit IRC03:47
*** Benj_ has joined #openstack-infra03:47
*** Benj_ has quit IRC03:48
*** ociuhandu has quit IRC03:51
*** baoli has joined #openstack-infra03:52
*** mbacchi has joined #openstack-infra03:56
*** baoli has quit IRC03:57
*** ociuhandu has joined #openstack-infra04:01
*** bexelbie has joined #openstack-infra04:04
*** ociuhandu has quit IRC04:06
*** zhurong has quit IRC04:10
*** thorst has joined #openstack-infra04:13
*** bexelbie has quit IRC04:13
*** thorst has quit IRC04:15
openstackgerritMasayuki Igawa proposed openstack/gertty: [WIP] Add trailing-whitespace style  https://review.openstack.org/36829304:16
*** bexelbie has joined #openstack-infra04:16
*** ociuhandu has joined #openstack-infra04:16
*** thorst has joined #openstack-infra04:18
*** thorst has quit IRC04:18
*** ociuhandu has quit IRC04:21
*** tphummel has joined #openstack-infra04:46
*** stewie925 has quit IRC04:51
*** skipp has quit IRC04:52
*** wcriswell has quit IRC04:52
*** eeiden has quit IRC04:53
*** Jeffrey4l_ has joined #openstack-infra05:00
*** Jeffrey4l has quit IRC05:04
*** zhurong has joined #openstack-infra05:05
*** _nadya_ has joined #openstack-infra05:06
*** wcriswell has joined #openstack-infra05:07
*** nijaba has quit IRC05:07
*** otherwiseguy has quit IRC05:07
*** hamzy has quit IRC05:07
*** khivin has quit IRC05:07
*** netsin has quit IRC05:07
*** pblaho has quit IRC05:07
*** nt has quit IRC05:07
*** zigo has quit IRC05:07
*** pcrews has quit IRC05:07
*** HenryG has quit IRC05:07
*** simonmcc has quit IRC05:07
*** ianychoi has quit IRC05:07
*** ediardo has quit IRC05:07
*** skath has quit IRC05:07
*** ameade has quit IRC05:07
*** tdurakov has quit IRC05:07
*** kozhukalov has quit IRC05:07
*** boris-42 has quit IRC05:07
*** igorbelikov has quit IRC05:07
*** monester_ has quit IRC05:07
*** zz_zz_ja has quit IRC05:07
*** docaedo has quit IRC05:07
*** amrith has quit IRC05:07
*** dstanek has quit IRC05:07
*** dimtruck has quit IRC05:07
*** timsim has quit IRC05:07
*** mrunge has quit IRC05:07
*** whoops has quit IRC05:07
*** dougwig has quit IRC05:07
*** jbryce has quit IRC05:07
*** lxsli has quit IRC05:07
*** Guest36298 has quit IRC05:07
*** calebb has quit IRC05:07
*** andreaf has quit IRC05:07
*** bethwhite has quit IRC05:07
*** lindsayk has quit IRC05:07
*** adreznec has quit IRC05:07
*** lewo has quit IRC05:07
*** mgagne has quit IRC05:07
*** khappone has quit IRC05:07
*** gtmanfred has quit IRC05:07
*** xarses has quit IRC05:07
*** pabelanger has quit IRC05:07
*** oomichi has quit IRC05:07
*** persia has quit IRC05:07
*** Swanson has quit IRC05:07
*** tobiash has quit IRC05:07
*** jtomasek has quit IRC05:07
*** jesusaur has quit IRC05:07
*** paulobanon has quit IRC05:07
*** Nakato has quit IRC05:07
*** vtapia has quit IRC05:07
*** jlvillal has quit IRC05:07
*** sc` has quit IRC05:07
*** cschwede has quit IRC05:07
*** jamespage has quit IRC05:07
*** johnsom has quit IRC05:07
*** yarkot1 has quit IRC05:07
*** htruta has quit IRC05:07
*** markmcclain has quit IRC05:07
*** Douhet has quit IRC05:07
*** notmyname has quit IRC05:07
*** aarefiev has quit IRC05:07
*** jd__ has quit IRC05:07
*** fernnest has quit IRC05:07
*** alex_xu has quit IRC05:07
*** timothyb89 has quit IRC05:07
*** ggillies has quit IRC05:07
*** sfinucan has quit IRC05:07
*** gibi has quit IRC05:07
*** katyafervent_awa has quit IRC05:07
*** rakhmerov has quit IRC05:07
*** igormarnat has quit IRC05:07
*** asilenkov has quit IRC05:07
*** flepied has quit IRC05:07
*** moravec has quit IRC05:07
*** devananda has quit IRC05:07
*** clenimar has quit IRC05:07
*** beekneemech has quit IRC05:07
*** mandre has quit IRC05:07
*** csmart has quit IRC05:07
*** akscram has quit IRC05:07
*** admcleod has quit IRC05:07
*** rwsu has quit IRC05:07
*** petems has quit IRC05:07
*** ianw has quit IRC05:07
*** Hunner has quit IRC05:07
*** gluytium has quit IRC05:07
*** prometheanfire has quit IRC05:07
*** rkukura has quit IRC05:07
*** toabctl has quit IRC05:07
*** mattoliverau has quit IRC05:07
*** clarkb has quit IRC05:07
*** mordred has quit IRC05:07
*** irtermite has quit IRC05:07
*** thcipriani has quit IRC05:07
*** nhandler has quit IRC05:07
*** abramley has quit IRC05:07
*** laurelm has quit IRC05:07
*** kaisers1 has quit IRC05:07
*** mwhahaha has quit IRC05:07
*** dkehn_ has quit IRC05:07
*** dkehn__ has quit IRC05:07
*** r-mibu has quit IRC05:07
*** YorikSar has quit IRC05:07
*** sanek has quit IRC05:07
*** rm_work has quit IRC05:07
*** krtaylor has quit IRC05:07
*** vaidy has quit IRC05:07
*** isviridov_away has quit IRC05:07
*** eharney has quit IRC05:07
*** kuromagi has quit IRC05:07
*** gus has quit IRC05:07
*** szaher has quit IRC05:07
*** acabot has quit IRC05:07
*** dgonzalez has quit IRC05:07
*** davidlenwell has quit IRC05:07
*** hogepodge has quit IRC05:07
*** AJaeger has quit IRC05:07
*** iremizov has quit IRC05:07
*** csatari has quit IRC05:07
*** d34dh0r53 has quit IRC05:07
*** cargonza has quit IRC05:07
*** bmjen has quit IRC05:07
*** peterlisak has quit IRC05:07
*** patrickeast has quit IRC05:07
*** dabukalam_ has quit IRC05:07
*** serverascode has quit IRC05:07
*** mjb has quit IRC05:07
*** rbradfor has quit IRC05:07
*** adam_g has quit IRC05:07
*** ryanpetrello has quit IRC05:07
*** craige has quit IRC05:07
*** ogelbukh has quit IRC05:07
*** madorn has quit IRC05:07
*** beisner has quit IRC05:07
*** cj has quit IRC05:07
*** JayF has quit IRC05:07
*** dhellmann has quit IRC05:07
*** med_ has quit IRC05:07
*** pleia2 has quit IRC05:07
*** odyssey4me has quit IRC05:07
*** alaski has quit IRC05:07
*** jamielennox has quit IRC05:07
*** lindsayk_ has quit IRC05:07
*** mhayden has quit IRC05:07
*** mrmartin has quit IRC05:07
*** mrhillsman has quit IRC05:07
*** Guest66666 has quit IRC05:07
*** dlahn has quit IRC05:07
*** charz has quit IRC05:07
*** vkmc has quit IRC05:07
*** tbarron has quit IRC05:07
*** alexm2 has quit IRC05:07
*** MarkAtwood has quit IRC05:07
*** knikolla has quit IRC05:07
*** vgadiraj has quit IRC05:07
*** SpamapS has quit IRC05:07
*** notmorgan has quit IRC05:07
*** BobBall has quit IRC05:07
*** fabo has quit IRC05:07
*** matbu has quit IRC05:07
*** russellb has quit IRC05:07
*** logan- has quit IRC05:07
*** thiagolib has quit IRC05:10
*** DuncanT has quit IRC05:10
*** _nadya_ has quit IRC05:11
*** thorst has joined #openstack-infra05:13
*** thiagolib has joined #openstack-infra05:18
*** rossella_s has quit IRC05:18
*** igorbelikov has joined #openstack-infra05:21
*** csatari has joined #openstack-infra05:21
*** DuncanT has joined #openstack-infra05:23
*** AnarchyAo has quit IRC05:23
*** ogelbukh has joined #openstack-infra05:25
*** nijaba has joined #openstack-infra05:25
*** otherwiseguy has joined #openstack-infra05:25
*** hamzy has joined #openstack-infra05:25
*** khivin has joined #openstack-infra05:25
*** netsin has joined #openstack-infra05:25
*** pblaho has joined #openstack-infra05:25
*** ianychoi has joined #openstack-infra05:25
*** nt has joined #openstack-infra05:25
*** zigo has joined #openstack-infra05:25
*** pcrews has joined #openstack-infra05:25
*** HenryG has joined #openstack-infra05:25
*** simonmcc has joined #openstack-infra05:25
*** skath has joined #openstack-infra05:25
*** ameade has joined #openstack-infra05:25
*** tdurakov has joined #openstack-infra05:25
*** boris-42 has joined #openstack-infra05:25
*** kozhukalov has joined #openstack-infra05:25
*** monester_ has joined #openstack-infra05:25
*** zz_zz_ja has joined #openstack-infra05:25
*** docaedo has joined #openstack-infra05:25
*** amrith has joined #openstack-infra05:25
*** dstanek has joined #openstack-infra05:25
*** dimtruck has joined #openstack-infra05:25
*** timsim has joined #openstack-infra05:25
*** mrunge has joined #openstack-infra05:25
*** whoops has joined #openstack-infra05:25
*** dougwig has joined #openstack-infra05:25
*** jbryce has joined #openstack-infra05:25
*** lxsli has joined #openstack-infra05:25
*** Guest36298 has joined #openstack-infra05:25
*** calebb has joined #openstack-infra05:25
*** andreaf has joined #openstack-infra05:25
*** bethwhite has joined #openstack-infra05:25
*** lindsayk has joined #openstack-infra05:25
*** adreznec has joined #openstack-infra05:25
*** lewo has joined #openstack-infra05:25
*** mgagne has joined #openstack-infra05:25
*** khappone has joined #openstack-infra05:25
*** gtmanfred has joined #openstack-infra05:25
*** ogelbukh has quit IRC05:25
*** ogelbukh has joined #openstack-infra05:25
*** petems has joined #openstack-infra05:25
*** DuncanT has quit IRC05:27
*** ameade has quit IRC05:28
*** ameade has joined #openstack-infra05:30
*** Mary has joined #openstack-infra05:32
*** Mary is now known as Guest9970505:32
*** tphummel has quit IRC05:33
*** Guest99705 has quit IRC05:37
*** askb has joined #openstack-infra05:39
*** askb has quit IRC05:42
*** eeiden has joined #openstack-infra05:45
*** rvba has quit IRC05:47
*** kaisers has joined #openstack-infra05:49
*** wcriswell has quit IRC05:50
*** eeiden has quit IRC05:50
*** gongysh has joined #openstack-infra05:52
*** wcriswell has joined #openstack-infra05:57
*** thorst has quit IRC06:04
*** rvba has joined #openstack-infra06:08
*** rvba has quit IRC06:08
*** rvba has joined #openstack-infra06:08
*** thorst has joined #openstack-infra06:13
*** thorst_ has joined #openstack-infra06:14
*** thorst has quit IRC06:18
*** thorst_ has quit IRC06:19
*** zhurong has quit IRC06:21
*** _nadya_ has joined #openstack-infra06:28
*** senk has joined #openstack-infra06:28
*** notmorgan has joined #openstack-infra06:32
*** stewie925 has joined #openstack-infra06:34
*** skipp has joined #openstack-infra06:35
*** thorst has joined #openstack-infra06:38
*** dimtruck is now known as zz_dimtruck06:41
*** kzaitsev_mb has joined #openstack-infra06:43
*** sdake_ has joined #openstack-infra06:45
*** sdake has quit IRC06:48
*** ilyashakhat has joined #openstack-infra06:49
*** e0ne has joined #openstack-infra06:53
*** _nadya_ has quit IRC06:58
*** thorst has quit IRC07:03
*** eeiden has joined #openstack-infra07:07
*** gongysh has quit IRC07:11
*** tqtran has joined #openstack-infra07:12
openstackgerritMerged openstack-infra/project-config: Pin bundler to 1.12.5 for some jobs  https://review.openstack.org/36826707:13
*** gouthamr has joined #openstack-infra07:14
*** e0ne has quit IRC07:14
*** tqtran has quit IRC07:16
*** e0ne has joined #openstack-infra07:17
openstackgerritMerged openstack-infra/project-config: Add new project networking-zte  https://review.openstack.org/35527807:18
*** _nadya_ has joined #openstack-infra07:20
*** gouthamr_ has joined #openstack-infra07:20
*** gouthamr has quit IRC07:24
*** amotoki has joined #openstack-infra07:26
*** tonytan4ever has quit IRC07:26
openstackgerritMerged openstack-infra/project-config: Add new projects: fuel-ccp-murano and fuel-ccp-sahara  https://review.openstack.org/36756107:28
*** hieulq has joined #openstack-infra07:30
*** Mary has joined #openstack-infra07:32
*** Mary is now known as Guest9379007:32
*** sdake_ has quit IRC07:35
*** e0ne has quit IRC07:35
*** Guest93790 has quit IRC07:37
*** ilyashakhat has quit IRC07:45
*** thorst has joined #openstack-infra07:47
*** _nadya_ has quit IRC07:47
*** amotoki has quit IRC07:49
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Cleanup gerrit/projects.yaml  https://review.openstack.org/36830607:50
*** thorst has quit IRC07:51
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** ihrachys has joined #openstack-infra08:04
*** zul has joined #openstack-infra08:04
*** DuncanT has joined #openstack-infra08:04
*** ediardo has joined #openstack-infra08:04
*** cargonza has joined #openstack-infra08:04
*** MarkAtwood has joined #openstack-infra08:04
*** michaelxin has joined #openstack-infra08:04
*** kaisers1 has joined #openstack-infra08:04
*** mwhahaha has joined #openstack-infra08:04
*** dkehn_ has joined #openstack-infra08:04
*** dkehn__ has joined #openstack-infra08:04
*** r-mibu has joined #openstack-infra08:04
*** YorikSar has joined #openstack-infra08:04
*** sanek has joined #openstack-infra08:04
*** rm_work has joined #openstack-infra08:04
*** krtaylor has joined #openstack-infra08:04
*** vaidy has joined #openstack-infra08:04
*** isviridov_away has joined #openstack-infra08:04
*** eharney has joined #openstack-infra08:04
*** kuromagi has joined #openstack-infra08:04
*** gus has joined #openstack-infra08:04
*** szaher has joined #openstack-infra08:04
*** dgonzalez has joined #openstack-infra08:04
*** davidlenwell has joined #openstack-infra08:04
*** hogepodge has joined #openstack-infra08:04
*** AJaeger has joined #openstack-infra08:04
*** iremizov has joined #openstack-infra08:04
*** d34dh0r53 has joined #openstack-infra08:04
*** bmjen has joined #openstack-infra08:04
*** peterlisak has joined #openstack-infra08:04
*** patrickeast has joined #openstack-infra08:04
*** serverascode has joined #openstack-infra08:04
*** dabukalam_ has joined #openstack-infra08:04
*** mjb has joined #openstack-infra08:04
*** rbradfor has joined #openstack-infra08:04
*** adam_g has joined #openstack-infra08:04
*** ryanpetrello has joined #openstack-infra08:04
*** craige has joined #openstack-infra08:04
*** madorn has joined #openstack-infra08:04
*** dlahn has joined #openstack-infra08:04
*** beisner has joined #openstack-infra08:04
*** cj has joined #openstack-infra08:04
*** JayF has joined #openstack-infra08:04
*** vgadiraj has joined #openstack-infra08:04
*** dhellmann has joined #openstack-infra08:04
*** med_ has joined #openstack-infra08:04
*** pleia2 has joined #openstack-infra08:04
*** odyssey4me has joined #openstack-infra08:04
*** alaski has joined #openstack-infra08:04
*** jamielennox has joined #openstack-infra08:04
*** lindsayk_ has joined #openstack-infra08:04
*** mhayden has joined #openstack-infra08:04
*** mrmartin has joined #openstack-infra08:04
*** mrhillsman has joined #openstack-infra08:04
*** Guest66666 has joined #openstack-infra08:04
*** knikolla has joined #openstack-infra08:04
*** charz has joined #openstack-infra08:04
*** vkmc has joined #openstack-infra08:04
*** tbarron has joined #openstack-infra08:04
*** alexm2 has joined #openstack-infra08:04
*** SpamapS has joined #openstack-infra08:04
*** BobBall has joined #openstack-infra08:04
*** fabo has joined #openstack-infra08:04
*** matbu has joined #openstack-infra08:04
*** russellb has joined #openstack-infra08:04
*** logan- has joined #openstack-infra08:04
*** serverascode has quit IRC08:04
*** DuncanT has quit IRC08:04
*** 17SAA4J1R has joined #openstack-infra08:05
*** rossella_s has joined #openstack-infra08:05
*** katyafervent_awa has joined #openstack-infra08:05
*** rakhmerov has joined #openstack-infra08:05
*** igormarnat has joined #openstack-infra08:05
*** asilenkov has joined #openstack-infra08:05
*** flepied has joined #openstack-infra08:05
*** moravec has joined #openstack-infra08:05
*** devananda has joined #openstack-infra08:05
*** clenimar has joined #openstack-infra08:05
*** beekneemech has joined #openstack-infra08:05
*** mandre has joined #openstack-infra08:05
*** csmart has joined #openstack-infra08:05
*** akscram has joined #openstack-infra08:05
*** admcleod has joined #openstack-infra08:05
*** rwsu has joined #openstack-infra08:05
*** ianw has joined #openstack-infra08:05
*** Hunner has joined #openstack-infra08:05
*** gluytium has joined #openstack-infra08:05
*** prometheanfire has joined #openstack-infra08:05
*** rkukura has joined #openstack-infra08:05
*** toabctl has joined #openstack-infra08:05
*** mattoliverau has joined #openstack-infra08:05
*** clarkb has joined #openstack-infra08:05
*** mordred has joined #openstack-infra08:05
*** irtermite has joined #openstack-infra08:05
*** thcipriani has joined #openstack-infra08:05
*** nhandler has joined #openstack-infra08:05
*** abramley has joined #openstack-infra08:05
*** laurelm has joined #openstack-infra08:05
*** yamahata has joined #openstack-infra08:05
*** xarses has joined #openstack-infra08:05
*** pabelanger has joined #openstack-infra08:05
*** oomichi has joined #openstack-infra08:05
*** johnsom has joined #openstack-infra08:05
*** Swanson has joined #openstack-infra08:05
*** persia has joined #openstack-infra08:05
*** tobiash has joined #openstack-infra08:05
*** jesusaur has joined #openstack-infra08:05
*** paulobanon has joined #openstack-infra08:05
*** Nakato has joined #openstack-infra08:05
*** vtapia has joined #openstack-infra08:05
*** jlvillal has joined #openstack-infra08:05
*** sc` has joined #openstack-infra08:05
*** cschwede has joined #openstack-infra08:05
*** jamespage has joined #openstack-infra08:05
*** yarkot1 has joined #openstack-infra08:05
*** htruta has joined #openstack-infra08:05
*** markmcclain has joined #openstack-infra08:05
*** Douhet has joined #openstack-infra08:05
*** notmyname has joined #openstack-infra08:05
*** aarefiev has joined #openstack-infra08:05
*** jd__ has joined #openstack-infra08:05
*** fernnest has joined #openstack-infra08:05
*** alex_xu has joined #openstack-infra08:05
*** timothyb89 has joined #openstack-infra08:05
*** ggillies has joined #openstack-infra08:05
*** sfinucan has joined #openstack-infra08:05
*** gibi has joined #openstack-infra08:05
*** kzaitsev_mb has quit IRC08:05
*** iremizov has quit IRC08:06
*** hrybacki has quit IRC08:07
*** MarkAtwood has quit IRC08:08
*** thorst has joined #openstack-infra08:08
*** petems has quit IRC08:08
*** serverascode has joined #openstack-infra08:11
*** 17SAA4J1R has quit IRC08:12
*** iremizov has joined #openstack-infra08:14
*** DuncanT has joined #openstack-infra08:14
*** hrybacki has joined #openstack-infra08:15
*** MarkAtwood has joined #openstack-infra08:16
*** thorst has quit IRC08:16
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/36831108:16
*** petems has joined #openstack-infra08:16
*** gouthamr_ has quit IRC08:17
*** hieulq has quit IRC08:18
*** senk has quit IRC08:22
*** senk has joined #openstack-infra08:23
*** tonytan4ever has joined #openstack-infra08:26
*** salv-orlando has joined #openstack-infra08:30
*** rcernin has joined #openstack-infra08:31
*** hieulq has joined #openstack-infra08:31
*** tonytan4ever has quit IRC08:32
*** Mary has joined #openstack-infra08:32
*** Mary is now known as Guest4911408:32
*** salv-orl_ has quit IRC08:33
*** HeOS has joined #openstack-infra08:36
*** Guest49114 has quit IRC08:37
*** thorst has joined #openstack-infra08:45
*** thorst has quit IRC08:49
openstackgerritMerged openstack-infra/tripleo-ci: Set default for pingtest template  https://review.openstack.org/36375308:52
openstackgerritHieu LE proposed openstack-infra/project-config: Add Magnum API Reference jobs template  https://review.openstack.org/36831308:56
*** e0ne has joined #openstack-infra08:58
*** Jeffrey4l_ has quit IRC09:02
*** ilyashakhat_mobi has joined #openstack-infra09:12
*** tqtran has joined #openstack-infra09:14
*** zul has quit IRC09:17
*** ilyashakhat_mobi has quit IRC09:17
*** tqtran has quit IRC09:19
*** ihrachys has quit IRC09:20
*** tobiash has quit IRC09:21
*** hieulq has quit IRC09:22
*** amotoki has joined #openstack-infra09:27
*** thorst has joined #openstack-infra09:30
*** r-mibu has quit IRC09:33
*** HeOS has quit IRC09:33
*** r-mibu has joined #openstack-infra09:34
*** thorst has quit IRC09:35
*** tobiash has joined #openstack-infra09:35
*** rcernin has quit IRC09:56
*** HeOS has joined #openstack-infra09:58
*** kaisers has quit IRC10:00
*** dosaboy has quit IRC10:02
*** kzaitsev_mb has joined #openstack-infra10:02
*** HeOS has quit IRC10:03
*** amotoki has quit IRC10:11
*** amotoki has joined #openstack-infra10:14
*** e0ne has quit IRC10:16
*** ilyashakhat_mobi has joined #openstack-infra10:21
*** ilyashakhat_mobi has quit IRC10:25
*** tonytan4ever has joined #openstack-infra10:27
*** ramishra has quit IRC10:29
*** tonytan4ever has quit IRC10:32
*** ramishra has joined #openstack-infra10:33
*** amotoki has quit IRC10:40
*** asettle has joined #openstack-infra10:50
*** ihrachys has joined #openstack-infra10:52
*** ihrachys has quit IRC10:56
*** amotoki has joined #openstack-infra10:56
*** amotoki has quit IRC11:09
*** kzaitsev_mb has quit IRC11:11
*** tqtran has joined #openstack-infra11:15
*** gongysh has joined #openstack-infra11:18
*** tqtran has quit IRC11:19
*** ilyashakhat_mobi has joined #openstack-infra11:21
*** ilyashakhat_mobi has quit IRC11:26
*** yamahata has quit IRC11:27
*** asettle has quit IRC11:45
*** rhallisey has joined #openstack-infra11:47
*** kaisers has joined #openstack-infra11:49
*** kaisers has quit IRC11:54
*** tonytan4ever has joined #openstack-infra11:58
*** tonytan4ever has quit IRC12:03
*** kzaitsev_mb has joined #openstack-infra12:07
*** asettle has joined #openstack-infra12:18
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove some TODOs from zuul/layout.yaml  https://review.openstack.org/36832212:18
*** kaisers has joined #openstack-infra12:23
AJaegerinfra-root, the following repos are not setup properly in github: https://github.com/openstack/deb-novnc https://github.com/openstack/deb-openstack-meta-packages https://github.com/openstack/deb-openstack-nose . Could somebody finish the setup, please?12:32
*** asettle has quit IRC12:37
*** bexelbie has quit IRC12:41
*** bexelbie has joined #openstack-infra12:41
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add missing deb repo 4/5  https://review.openstack.org/36673412:47
*** amotoki has joined #openstack-infra12:54
*** tonytan4ever has joined #openstack-infra12:59
*** ilyashakhat_mobi has joined #openstack-infra13:00
*** Jeffrey4l_ has joined #openstack-infra13:01
*** tonytan4ever has quit IRC13:04
*** amotoki has quit IRC13:11
*** zhurong has joined #openstack-infra13:13
*** tqtran has joined #openstack-infra13:17
fungiAJaeger: odd, i fixed permission on all three. deb-openstack-nose showed up right away, deb-openstack-meta-packages didn't until i used the gerrit api to force replication start for it again, and deb-novnc still hasn't shown up. maybe that one just needs more patience13:18
*** tqtran has quit IRC13:21
*** zhurong has quit IRC13:22
fungiand gerrit show-queue doesn't indicate there's any replication hung/waiting for it either13:22
openstackgerritMerged openstack-infra/puppet-openstack_infra_spec_helper: Pin puppetlabs_spec_helper ~> 1.1.1  https://review.openstack.org/36825413:28
*** fguillot has joined #openstack-infra13:29
*** fguillot has quit IRC13:30
fungioops, i did not mean to merge that one. i'll revert13:31
openstackgerritJeremy Stanley proposed openstack-infra/puppet-openstack_infra_spec_helper: Revert "Pin puppetlabs_spec_helper ~> 1.1.1"  https://review.openstack.org/36832713:34
openstackgerritJeremy Stanley proposed openstack-infra/puppet-openstack_infra_spec_helper: Revert "Pin puppetlabs_spec_helper ~> 1.1.1"  https://review.openstack.org/36832713:35
*** fguillot has joined #openstack-infra13:38
*** fguillot has quit IRC13:38
*** fguillot has joined #openstack-infra13:39
openstackgerritJeremy Stanley proposed openstack-infra/puppet-mediawiki: Skip CAPTCHA for autopatrol users  https://review.openstack.org/36811413:39
openstackgerritMerged openstack-infra/infra-specs: Prioritize Docs Publishing via AFS  https://review.openstack.org/36630313:42
*** ddieterly has joined #openstack-infra13:43
openstackgerritMerged openstack-infra/puppet-openstack_infra_spec_helper: Revert "Pin puppetlabs_spec_helper ~> 1.1.1"  https://review.openstack.org/36832713:45
*** Goneri has joined #openstack-infra13:49
*** sdake has joined #openstack-infra13:52
*** gongysh has quit IRC13:56
*** tonytan4ever has joined #openstack-infra14:00
*** ddieterly has quit IRC14:01
openstackgerritMerged openstack-infra/puppet-mediawiki: Skip CAPTCHA for autopatrol users  https://review.openstack.org/36811414:01
*** tonytan4ever has quit IRC14:04
*** zul has joined #openstack-infra14:06
*** zul has quit IRC14:07
*** ddieterly has joined #openstack-infra14:11
*** ddieterly has quit IRC14:13
*** salv-orl_ has joined #openstack-infra14:30
*** salv-orlando has quit IRC14:33
*** ddieterly has joined #openstack-infra14:37
openstackgerritJeremy Stanley proposed openstack-infra/puppet-mediawiki: Make robots.txt reachable  https://review.openstack.org/36833114:38
*** ddieterly has quit IRC14:42
openstackgerritMerged openstack-infra/storyboard-webclient: Fix behaviour for logged-out users  https://review.openstack.org/36817614:42
*** ddieterly has joined #openstack-infra14:44
*** ddieterly has quit IRC14:47
*** ddieterly has joined #openstack-infra14:49
*** Goneri has quit IRC14:50
*** sdake_ has joined #openstack-infra15:04
*** sdake has quit IRC15:06
*** timello has quit IRC15:08
*** tonytan4ever has joined #openstack-infra15:12
*** timello has joined #openstack-infra15:18
*** tqtran has joined #openstack-infra15:18
*** armax has joined #openstack-infra15:21
*** tqtran has quit IRC15:23
openstackgerritMerged openstack-infra/puppet-mediawiki: Make robots.txt reachable  https://review.openstack.org/36833115:27
*** aviau has quit IRC15:35
*** aviau has joined #openstack-infra15:35
*** ddieterly has quit IRC15:37
*** armax has quit IRC15:44
*** kaisers has quit IRC15:46
*** fguillot has quit IRC15:54
*** ddieterly has joined #openstack-infra15:56
*** ilyashakhat_mobi has quit IRC16:05
*** thorst has joined #openstack-infra16:06
*** kaisers has joined #openstack-infra16:10
AJaegerfungi, something is wrong with novnc - http://git.openstack.org/cgit/openstack/deb-novnc is empty as well, so the initial import did not even work ;(16:14
mordredfungi: for some reasons our trusty puppet jobs are trying (and failing, obvs) to hit ec2 metadata16:14
mordredfungi: http://logs.openstack.org/39/368239/3/check/gate-infra-puppet-apply-ubuntu-trusty/c4ada48/console.html#_2016-09-10_15_20_41_29254616:14
mordredfungi: I'm not sure what's up with that - maybe related to the bundler stuff?16:15
*** jpeeler has quit IRC16:15
*** fguillot has joined #openstack-infra16:15
AJaegerhi mordred !16:15
fungiAJaeger: yep, i mentioned that earlier16:16
fungiAJaeger: oh, empty on git.o.o too?16:16
AJaegerfungi, what I mean: empty on git.o.o16:16
AJaegerfungi, yes.16:16
AJaegerso, the problem is earlier...16:16
fungiAJaeger: agreed16:16
fungiimport didn't work for some reason16:17
fungithat certainly explains why nothing was replicated to github16:17
* AJaeger only checks that github is ready and assumed that if that works, the import is fine - and if not, in 99 per cent it's a github issue, so didn't dig further initially16:17
AJaegeruntil now ;)16:17
AJaegermordred: could you help fixing deb-nodepool, please? https://review.openstack.org/36825616:18
*** toabctl has quit IRC16:18
AJaegerfungi, the novnc repo is not urgent IMHO - this could wait until Monday as well. Just wanted to have it out here16:19
fungimordred: yeah, that metadata failure is a new one on me. i doubt it's related to the new bundler release though16:20
fungimordred: is it happening consistently?16:21
AJaegerproject-config cores, I have two cleanup changes - fixing some nits: https://review.openstack.org/368306 and https://review.openstack.org/368322 . Review should be brief - but not urgent either. Just that this touches a few places and thus I'd like to avoid rebases16:21
AJaegerSo, reviews are welcome, please16:21
fungimordred: looks more like something facter is trying to do16:21
clarkbfungi: mordred facter does that16:21
fungiis the problem that it breaks if it gets a 5xx error but not if the metadata server is completely unreachable?16:22
fungii wonder if there's a way to configure facter to not try that at all16:23
AJaegerfungi, http://logs.openstack.org/06/368306/1/check/gate-project-config-gerrit/22a3fde/console.html#_2016-09-10_07_51_49_527246 - "master does not exist"16:23
AJaegerI wonder why that didn't fail?16:23
* AJaeger investigates the non-failing16:23
fungiAJaeger: maybe we've broken our exit code passthrough?16:23
AJaegermy guess as well, let's see16:24
amrithfungi, AJaeger, clarkb ... could one of you help me understand what ails https://review.openstack.org/#/c/368230/. I'm thinking it should be merged but it says "Needs verified" but I see no jobs in the zuul status even though it just passed a recheck.16:24
amrithit depends on nothing16:24
amrithso I'm puzzled what has become of it16:25
fungiamrith: https://review.openstack.org/288712 (its parent) can't be enqueued16:25
AJaegerfungi, no that is the first check for our own repo.16:26
AJaegerTo see whether it is created.16:26
fungiamrith: once 288712 passes check and gets enqueued into the gate, 368230 will get dragged in along with it16:26
amrithfungi, thanks. did not catch that. I guess peterstac didn't intend to do that but he was in the wrong git context when he did this. I'll rebase his change onto master and resubmit. thanks16:26
amriththanks fungi I was only looking for Depends-On and didn't look at parent.16:27
*** ddieterly has quit IRC16:27
AJaegerfungi, I can clone https://anonscm.debian.org/git/openstack/deb-novnc.git and it has a master branch. But:16:28
AJaegergit branch -a16:28
AJaeger* debian/newton16:28
AJaeger  remotes/origin/HEAD -> origin/debian/newton16:28
AJaeger  remotes/origin/debian/newton16:28
AJaeger  remotes/origin/master16:28
*** mhickey__ has joined #openstack-infra16:28
AJaegerSo, that looks odd to me16:29
AJaegerwhy is debian/newton the default?16:29
AJaegeryes, indeed - HEAD points to debian/newton16:30
AJaegerthat shouldn't be the case...16:30
*** amotoki has joined #openstack-infra16:32
AJaegerfungi, is that the problem? Then I enhance our tests to catch that...16:33
*** tonytan4ever has quit IRC16:33
*** fguillot has quit IRC16:33
fungiAJaeger: i'm not entirely sure, to be honest16:36
fungii mean, we normally expect head to point to master, but i don't recall whether having it reference a different branch breaks our imports16:37
*** apetrich has quit IRC16:38
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Improve gerrit checks  https://review.openstack.org/36833516:41
AJaegerfungi, I coded that now in ^ - and some other checks16:41
AJaegerLet me WIP for now - until some infra-root figured out why the import failed16:42
*** apetrich has joined #openstack-infra16:42
*** fguillot has joined #openstack-infra16:42
*** amotoki has quit IRC16:45
* AJaeger weekends again and wishes everybody a nice one as well...16:49
*** mhickey__ has quit IRC16:50
fungithanks AJaeger! you too17:05
mordredclarkb: yah - I just thought we'd disabled facter trying to do that. I guess I was wrong17:07
mordredfungi, AJaeger: sorry - got distracted by the weekend - do you still need help with anyhting?17:07
fungimordred: nah, i'm just trying to get caught up on wiki configuration/testing17:08
*** gouthamr has joined #openstack-infra17:15
*** gouthamr has quit IRC17:16
*** tqtran has joined #openstack-infra17:20
*** tqtran has quit IRC17:24
*** thorst has quit IRC17:30
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Conveniently link /root/.my.cnf on wiki servers  https://review.openstack.org/36833717:30
*** Sukhdev has joined #openstack-infra17:31
*** gouthamr has joined #openstack-infra17:35
*** flepied has quit IRC17:41
openstackgerritJeremy Stanley proposed openstack-infra/puppet-mediawiki: Allow sufficient access to serve image files  https://review.openstack.org/36833817:46
*** mbacchi has quit IRC17:49
fungiour file content for wiki.openstack.org is now being served from a cinder volume, and the https://wiki-dev.openstack.org/ is now working-ish with a snapshot of our database and files from production17:50
Krenairbroken ssl cert :/17:51
Krenairself-signed17:51
funginot broken, just self-signed17:51
fungiit's a test site17:51
*** Qiming has quit IRC17:51
Krenairyes broken, it's got the wrong CN17:51
fungiahh, fair enough, it has wiki-dev01 not wiki-dev17:52
Krenairhave you guys considered using Let's Encrypt?17:52
fungiit comes up from time to time. we'd rather not for a number of reasons17:52
Krenairinteresting. mind if I ask what the reasons are?17:52
fungimostly having to renew certs 10x as often means things risk breaking 10x as often. most of us are uncomfortable with an automated daemon replacing certs in the background when nobody's paying attention17:53
fungiand i've already seen my share of sites that decided to switch to let's encrypt spontaneously break 3 months later17:54
Krenairthat can happen if you set it up wrong and it can't/doesn't attempt to auto-renew17:54
fungiindeed17:54
fungie.g., fragile17:54
fungimore oversight needed17:55
fungiet cetera17:55
*** tqtran has joined #openstack-infra17:55
fungialso, personally i find the situation under which le arose detestable, after years of cacert.org getting the runaround from mofo, yet then they turn around and okay their own non-community-managed free cert vending machine with their cabforum owners17:55
*** dave4925_ has joined #openstack-infra17:56
fungithough that's just me, i certainly won't say our project shouldn't rely on it on those grounds17:56
*** Qiming has joined #openstack-infra17:56
Krenairthat sounds like a legitimate objection to me17:57
fungianyway, it's mostly that we order our production certs for a 2-year validity currently, and would need to replace them 6x a year rather than once every 2 years17:57
fungiand so far the automation around le for renewing certs in the background has shown itself to still be immature17:58
fungiif we only had a few certs, that would be one thing, but right now we have enough that we would exceed le's maximum renewals per month limit17:59
*** flepied has joined #openstack-infra17:59
fungiso we'd need some extra logic to stagger batches17:59
fungiand some scheduling of humans to check behind it at least until we got more comfortable it was working reliably18:00
Krenairyou give your instances like the wiki their own public IP instead of putting them behind a reverse proxy?18:00
*** tqtran has quit IRC18:00
fungiyeah, reverse proxies would add even more single points of failure18:00
fungiespecially since we'd need to put them in each location where we run servers (we18:00
fungi're trying to get away from running them all in one provider)18:01
*** gouthamr has quit IRC18:01
fungior else we'd have to rely on a third-party proxy service we don't control18:01
openstackgerritMerged openstack-infra/system-config: Conveniently link /root/.my.cnf on wiki servers  https://review.openstack.org/36833718:03
*** tonytan4ever has joined #openstack-infra18:04
fungithe only place we're running a reverse proxy on a separate server at the moment is git.openstack.org, and that's so we can distribute sessions across multiple git servers18:05
fungibut they're all in the same network, and all hiding behind one fqdn18:05
*** markvoelker has quit IRC18:06
Krenairno public IP probably would've also limited that issue with elasticsearch to the private network18:07
fungisort of. we still need to be able to log into the server18:07
fungino public ip on the server would mean proxying all our management traffic to it (ssh, snmp, et cetera) not just http(s)18:08
fungiwe don't have a shadow management net between our servers. they're hosted in public virtual server hosting providers18:08
fungiwe've discussed setting up an ipsec mesh between them, but that'18:09
fungis a lot of extra effort for questionable gain at the moment18:09
fungiand would still need something with public addresses to route between systems on disparate rfc-1918 networks18:10
pabelangerHmm: ImportError: No module named wrapt18:10
*** tonytan4ever has quit IRC18:10
pabelangerwhen starting nodepoold locally18:10
pabelangerhttp://paste.openstack.org/show/570953/18:10
clarkbfwiw if we feel that is desireable es can be co figured to listen on localhost instead18:10
fungiyeah, for cirrussearch's purposes, having es only listen on the loopback is likely sufficient18:11
fungi(in addition to the fact that under normal circumstances we apply iptables rules only allowing the protocols we intend)18:11
*** tqtran has joined #openstack-infra18:11
*** Sukhdev has quit IRC18:11
pabelangernotmorgan: ^ any ideas why wrapt wasn't installed?18:13
fungithe exposure in question happened due to an unfortunate combination of playing with virtual server snapshots, disabling configuration management, and performing an in-place distro release upgrade. three things we generally avoid18:13
pabelangerHmm, something failed with my install18:15
pabelangernotmorgan: ignore for now, I think it pebkac18:15
*** ddieterly has joined #openstack-infra18:16
*** yamahata has joined #openstack-infra18:18
pabelangerokay, using an older version of pip18:18
pabelangerthat explains it18:18
*** tqtran has quit IRC18:19
*** zz_dimtruck is now known as dimtruck18:21
fungiKrenair: anyway, snakeoil cert on wiki-dev.o.o site updated to match the vhost name18:21
*** tqtran has joined #openstack-infra18:22
*** gouthamr has joined #openstack-infra18:22
*** ddieterly has quit IRC18:23
fungilater this weekend or early in the week i'll see if i can spot what needs tweaking to get openid set up, since https://wiki-dev.openstack.org/wiki/Special:Version indicates it's installed but i don't see the login link on the page headers yet18:24
*** mriedem has joined #openstack-infra18:25
*** thorst has joined #openstack-infra18:30
*** mbacchi has joined #openstack-infra18:30
*** thorst has quit IRC18:35
*** akshai has joined #openstack-infra18:39
*** fguillot has quit IRC18:48
*** fguillot has joined #openstack-infra18:52
*** ociuhandu has joined #openstack-infra19:02
*** dimtruck is now known as zz_dimtruck19:02
*** flepied1 has joined #openstack-infra19:05
*** flepied has quit IRC19:05
*** kaisers has quit IRC19:10
*** apetrich has quit IRC19:16
Krenairfungi, one issue I notice is 403s for images and skin files19:17
*** tqtran has quit IRC19:19
fungiKrenair: yeah, https://review.openstack.org/368338 should solve that? i manually tested and it seemed to work, but puppet reverted it on the next pass of course19:19
Krenairyou could puppet agent --disable19:20
fungiwell, not actually since we don't use puppet agent19:20
Krenairoh right19:20
fungiwe have other central mechanisms for disabling puppet19:20
fungihowever, i'd rather see our automation actually working for this19:20
fungibut anyway, if 368338 looks safe to you i'll go ahead and approve it now19:21
sdake_clarkb ffungi - just wanted to give you heads up regarding that emergency merge- thanks for doing that19:22
sdake_our jobs wer etiming out becuse our gate jobs entirely lockekd up19:22
sdake_we still need the 90 minutes for awhile19:22
sdake_but the 90 minutes really helped me understand that we actually had a bugin our code causing the problem19:23
sdake_I want to thank infra for woring with us on that :)19:23
Krenairfungi, looks good to me, but I don't have approval rights19:23
*** senk has quit IRC19:23
sdake_on a side note, rax.org seems defective (or our use thereof)19:24
sdake_rax.ord19:24
sdake_it is a recent change19:24
fungisdake_: do you have any example failures from the ord region demonstrating the issue you're only seeing there?19:27
sdake_note we still need the centos mirrors:19:27
sdake_[12:26:21]  <Jeffrey4l_>centos binary  21 minuts to build + 18 min to deploy      ubuntu binary 8 min to build + 30 min to (deploy+reconfig+upgrade)19:27
sdake_fungi i wish i bookmakred it19:27
sdake_fungi but i dont have any idea where it is - but its consistent19:27
sdake_we ar emostly gttting sheulded to osic-cloud119:28
sdake_rax.ord is not scheduled often19:28
fungiyeah, osic-cloud1 has a lot more quota than our other providers currently19:28
sdake_understood19:28
sdake_when i see the next rax.ord failure - i'll let you know19:28
sdake_its somtehing ike "ssh can't connect to ip"19:28
sdake_in our gate scripts19:29
sdake_where the ip is teh nodepool id we use19:29
sdake_you may or  may not recall we had to hack around a nodepool bug t one pint19:29
sdake_my speculation is that the ip being put in the nodepool files have changed19:30
sdake_(or its always been a problem and ew didn't notice it ;)19:30
fungiit's possible the images in ord are older than in other places due to upload failures, or it could have to do with local networking specific to rackspace's ord region19:30
clarkbsdake_: we have centos mirrors... are you using them yet?19:30
sdake_clarkb - we need rdo mirrors19:31
sdake_clarkb but to answer your question more specifically we are not - but ptach is working its way through iteration19:31
sdake_most of our build time is spent pulling rdo packages19:31
openstackgerritMerged openstack-infra/puppet-mediawiki: Allow sufficient access to serve image files  https://review.openstack.org/36833819:32
sdake_of which there is no mirror19:32
sdake_clarkb part of the problem and i wasn't aware of this - people added more tests to the gate19:33
sdake_which extends us out psst an hour19:33
*** thorst has joined #openstack-infra19:33
fungiif there's a network problem specific to rax-ord nodes, maybe it's related to the local routes for the eth1 interfaces there? i see these on a sample one i just logged into: 10.176.0.0/12, 10.208.0.0/12, 10.211.0.0/1919:33
sdake_ya my memory thinks it was 10.211.something19:34
clarkbfungi the dns record for mirror should have thr public ip though19:34
fungiyep, the eth1 interface on the one i just looked at was 10.211.0.4/1919:34
sdake_fungi rther then guess - i'll make sure to bookmark the next build failure and g et back to you19:34
clarkbyou can ask logstash for that data too fwiw19:35
fungiclarkb: oh, i didn't think sdake_ was saying this was causing a mirror problem19:35
fungijust a "consistent failure for kolla jobs running in rax-ord"19:35
sdake_no it causes a gate job  failure19:35
clarkbfungi oh maybr I misparsed19:35
sdake_fungi right19:35
clarkbI combined the two issurs19:35
fungithe mirror discussion was separate, yeah19:35
clarkbbut they are distinct19:35
sdake_clarkb sorry about that :)19:35
sdake_yes orthogonal issues19:35
*** esikache1 has joined #openstack-infra19:36
sdake_ansible uses ssh on one of the interfaces19:36
sdake_it is possible it is using t he wrong one19:36
sdake_(in rax.ord)19:37
*** thorst has quit IRC19:37
sdake_https://github.com/openstack/kolla/blob/master/tools/setup_gate.sh#L69-L7419:38
*** esikachev has joined #openstack-infra19:38
fungithat sounds more like a failure to properly abstract the network usage in your jobs away from the system interfaces (which lack consistency between providers)19:38
sdake_the sad part is i think i wrote that and i'm not even sure what it does :(19:38
fungican you create virtual interfaces with predictable names/addresses and use those in your tests?19:39
sdake_fungi not really a netorking dude, but i'll ask the core team when they pop in19:39
fungicool. fwiw i'm probably not around much the rest of the weekend, but it can presuambly wait until monday19:40
fungior others in here can chime in19:40
sdake_yup19:40
sdake_just reporting in a thank you for openstack-infra's support :)19:40
*** esikache1 has quit IRC19:41
fungialways glad to help19:41
clarkbsdake_: if your tests dont already run in parallel you might consider making that change to speed things up19:43
*** esikachev has quit IRC19:43
clarkbit has worked well for us getting the most out of these cpus. also you shake loose more fun bugs19:43
sdake_clarkb - our tests are deploy - reconfigure - upgrade - must run serially19:44
sdake_sometimes this operation fails on rax.org (but didn't in this case): http://logs.openstack.org/42/368342/1/check/gate-kolla-dsvm-deploy-ubuntu-binary-ubuntu-xenial-nv/2bc9c4b/console.html#_2016-09-10_18_25_31_25993619:45
*** esikachev has joined #openstack-infra19:45
clarkbsdake_: but you run a testsuite with test cases iirc19:45
clarkbat least thats what it looked like in the job logs I was reading19:45
sdake_clarkb just to build images - and all that does is run a python build script (which uses threads and parallelism)19:46
sdake_the remainder of our tests must be serial19:46
sdake_speaking of that - we probably need a lock file in our code..19:46
* sdake_ groans19:46
*** sdake_ is now known as sdake19:47
sdakeclarkb we test all of  our external apis now in the gate essentially - which is why our gate performance exceeds 1 hour in some cases19:49
*** Goneri has joined #openstack-infra19:49
sdakeclarkb just wait until next cycle - we are doing multtinode gating ;)19:53
sdakemainly to get compute on a separate vm19:53
sdakeour control node can run well in 8gb ram19:53
sdakebut control + compute in 8gig = tight19:54
wznoinskhi19:54
sdakealtough what we have now owrks most of the time19:54
sdakesup wznoinsk19:54
wznoinskjust wondering, should https://review.openstack.org/#/c/366352/ make this populated http://git.openstack.org/cgit/openstack/deb-novnc?19:54
wznoinskhi sdake19:54
*** fguillot has quit IRC19:55
*** mhickey__ has joined #openstack-infra19:55
* wznoinsk should probably check the code himself, just need to make the dinner for kids at this very moment19:56
sdakewell thanks folks19:57
sdakei'm out for a bit19:57
*** fguillot has joined #openstack-infra20:00
clarkbsdake: ya its the api testing you should br able to do in parallel just like tempest20:00
clarkbwznoinsk: its a knowm issue but cause hasnt been figured out20:01
sdakeclarkb nah - if you reconfigure at the same time as upgrade -  bad things happen20:01
clarkbsdake: right but thats true of serial or parallel api tests20:01
sdakeclarkb one way to think about it is upgrade changes all running openstack services20:01
*** fguillot has quit IRC20:01
sdakeand reconfigure sort of  does the same thing20:01
sdakeright - we need a lock file to prevent those operations from running in palralel20:02
sdakein case an op decides to login twice and get crazy with kolla :)20:02
sdakenaturally these two operations race against each other and cause all kinds of chaos20:02
sdakekolla is not designed to handle palrall operations of these apis20:02
*** markvoelker has joined #openstack-infra20:07
*** ilyashakhat_mobi has joined #openstack-infra20:07
*** markvoelker has quit IRC20:11
*** coolsvap has quit IRC20:12
*** amitgandhinz has quit IRC20:13
*** rakhmerov has quit IRC20:13
*** amitgandhinz has joined #openstack-infra20:13
*** igormarnat has quit IRC20:14
*** igormarnat has joined #openstack-infra20:17
*** rakhmerov has joined #openstack-infra20:17
*** katyafervent_awa has quit IRC20:21
*** asilenkov has quit IRC20:23
Krenairfungi, that seems to have fixed images20:23
fungicool. my original testing indicated that it should, just an oversight in earlier versions of the vhost template20:24
*** Goneri has quit IRC20:24
Krenairstill got the skins thing to do20:24
*** katyafervent_awa has joined #openstack-infra20:25
*** asilenkov has joined #openstack-infra20:25
Krenairand another issue showing up is https://meta.wikimedia.org/geoiplookup returning 404 (we shut that service down)20:25
KrenairI think that's called by UniversalLanguageSelector20:27
*** tqtran has joined #openstack-infra20:28
Krenairseems it can be disabled with $wgULSGeoService = false20:29
*** salv-orlando has joined #openstack-infra20:30
*** flepied1 is now known as flepied20:31
*** tqtran has quit IRC20:32
*** salv-orl_ has quit IRC20:33
*** salv-orl_ has joined #openstack-infra20:34
*** salv-orlando has quit IRC20:34
wznoinskclarkb: I saw a couple h lag between governance change merged and the git repository actually populated, is that a different issue we see here?20:40
clarkbwznoinsk: yes there is no data in gerrit iirc20:41
clarkbI wasnt looking into it though fungi and AJaeger were20:41
fungii wasn't really looking into it per se. manage-projects failed to import the repo, AJaeger suggested that might be due to HEAD pointing at the debian/newton branch rather than the master branch (though the imported repo did at least have a master branch)20:43
fungii suppose we could revert its inclusion in gerrit/projects.yaml until someone has a chance to go hunting in manage-projects and gerrit service logs20:44
*** ilyashakhat_mobi has quit IRC20:51
*** Goneri has joined #openstack-infra20:52
*** kaisers has joined #openstack-infra20:59
*** mhickey__ has quit IRC20:59
*** esikachev has quit IRC20:59
*** kaisers has quit IRC21:04
*** tonytan4ever has joined #openstack-infra21:04
*** Jeffrey4l_ has quit IRC21:07
*** tonytan4ever has quit IRC21:09
*** fguillot has joined #openstack-infra21:09
*** senk has joined #openstack-infra21:28
*** senk has quit IRC21:52
*** Jeffrey4l has joined #openstack-infra21:56
*** rhallisey has quit IRC21:58
*** markvoelker has joined #openstack-infra22:08
*** markvoelker has quit IRC22:12
*** tqtran has joined #openstack-infra22:13
*** rhallisey has joined #openstack-infra22:13
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Temporarily remove deb-novnc from projects.yaml  https://review.openstack.org/36836022:17
*** tqtran has quit IRC22:17
fungiclarkb: wznoinsk: AJaeger: zigo: i'm going to self-approve that ^22:17
*** askb has joined #openstack-infra22:19
wznoinskfungi:  +122:19
*** Goneri has quit IRC22:22
*** harlowja has quit IRC22:23
*** Goneri has joined #openstack-infra22:25
*** rhallisey has quit IRC22:25
*** csatari has quit IRC22:27
*** csatari has joined #openstack-infra22:29
*** Daviey_ has joined #openstack-infra22:33
*** ddieterly has joined #openstack-infra22:37
*** amit213_ has joined #openstack-infra22:37
*** Daviey has quit IRC22:37
*** amit213 has quit IRC22:37
*** rbergeron has quit IRC22:37
*** rbergero1 has joined #openstack-infra22:37
*** bstinson_ has joined #openstack-infra22:37
*** bstinson has quit IRC22:37
*** amit213_ is now known as amit21322:38
*** kzaitsev_mb has quit IRC22:38
*** kzaitsev_mb has joined #openstack-infra22:39
*** gouthamr has quit IRC22:39
*** winggundamth_ has joined #openstack-infra22:44
*** ddieterly has quit IRC22:45
openstackgerritMerged openstack-infra/project-config: Temporarily remove deb-novnc from projects.yaml  https://review.openstack.org/36836022:45
*** winggundamth has quit IRC22:45
*** kaisers has joined #openstack-infra22:48
*** kaisers has quit IRC22:53
*** esikachev has joined #openstack-infra22:53
*** esikachev has quit IRC22:59
*** tdasilva has quit IRC23:07
*** ddieterly has joined #openstack-infra23:07
*** sdake has quit IRC23:08
*** briancline has quit IRC23:10
*** tdasilva has joined #openstack-infra23:10
*** briancline has joined #openstack-infra23:10
*** rhallisey has joined #openstack-infra23:27
*** ddieterly has quit IRC23:34
*** kzaitsev_mb has quit IRC23:41
*** jcoufal has joined #openstack-infra23:48

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!