Monday, 2016-03-28

*** akshai has quit IRC00:02
*** achanda has quit IRC00:03
*** darvon has quit IRC00:04
*** cfarquhar has joined #openstack-infra00:07
*** cfarquhar has quit IRC00:07
*** cfarquhar has joined #openstack-infra00:07
*** darvon has joined #openstack-infra00:12
*** asettle has quit IRC00:27
*** IlyaG has joined #openstack-infra00:29
*** baoli has joined #openstack-infra00:31
*** IlyaG has quit IRC00:33
*** baoli has quit IRC00:34
*** kzaitsev_mb has joined #openstack-infra00:37
*** hichihara has joined #openstack-infra00:39
*** kzaitsev_mb has quit IRC00:42
*** zz_dimtruck is now known as dimtruck00:43
*** jerryz has quit IRC00:46
*** thorst has joined #openstack-infra00:53
*** amitgandhinz has joined #openstack-infra00:55
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add migration to move attrs and tags into columns
*** thorst has quit IRC01:01
*** achanda has joined #openstack-infra01:01
*** amitgandhinz has quit IRC01:09
*** piet has joined #openstack-infra01:09
*** zhurong has joined #openstack-infra01:10
openstackgerritwatanabe isao proposed openstack-infra/devstack-gate: Let BUILD_TIMEOUT in Ironic local.conf configurable
*** yamamoto has joined #openstack-infra01:16
*** mriedem has quit IRC01:19
*** zhurong has quit IRC01:23
*** baoli has joined #openstack-infra01:24
*** zhurong has joined #openstack-infra01:24
*** sabari has quit IRC01:24
*** sc68cal has quit IRC01:27
*** yamamoto has quit IRC01:28
*** hichihara has quit IRC01:29
*** hichihara has joined #openstack-infra01:29
*** Sukhdev has joined #openstack-infra01:38
*** kzaitsev_mb has joined #openstack-infra01:38
*** sabari- has joined #openstack-infra01:38
*** achanda has quit IRC01:39
*** piet has quit IRC01:40
*** piet has joined #openstack-infra01:41
*** mylu has quit IRC01:42
*** mylu has joined #openstack-infra01:43
*** kzaitsev_mb has quit IRC01:43
*** IlyaG has joined #openstack-infra01:45
*** tphummel has joined #openstack-infra01:46
*** IlyaG has quit IRC01:49
*** chuck_ has joined #openstack-infra01:50
*** chuck_ has quit IRC01:50
*** cfarquhar has quit IRC01:52
*** Qiming has joined #openstack-infra01:52
*** ZZelle has quit IRC01:53
*** piet has quit IRC01:54
*** zhurong has quit IRC01:55
*** tphummel has quit IRC01:56
*** thorst has joined #openstack-infra01:58
*** kzaitsev_mb has joined #openstack-infra02:03
*** Daisy has joined #openstack-infra02:03
*** amitgandhinz has joined #openstack-infra02:05
*** thorst has quit IRC02:06
*** baoli has quit IRC02:06
*** moravec has joined #openstack-infra02:10
openstackgerritHieu LE proposed openstack-infra/groups-static-pages: Update Vietnam User Group info
*** sdake has joined #openstack-infra02:11
*** baoli has joined #openstack-infra02:13
*** jamielennox is now known as jamielennox|away02:13
openstackgerritHieu LE proposed openstack-infra/groups-static-pages: Update Vietnam User Group info
*** sdake has quit IRC02:18
*** amitgandhinz has quit IRC02:19
*** kzaitsev_mb has quit IRC02:21
*** achanda has joined #openstack-infra02:26
*** yamamoto has joined #openstack-infra02:29
*** baoli has quit IRC02:32
*** gmann has quit IRC02:39
*** baoli has joined #openstack-infra02:41
*** Sukhdev has quit IRC02:42
*** mhayden has quit IRC02:43
*** mhayden has joined #openstack-infra02:43
*** gmann has joined #openstack-infra02:44
*** baoli has quit IRC02:44
*** Sukhdev has joined #openstack-infra02:50
*** Sukhdev has quit IRC02:51
*** mhayden has quit IRC02:51
*** baoli has joined #openstack-infra02:52
*** baoli has quit IRC02:54
*** cfarquhar has joined #openstack-infra02:55
*** cfarquhar has quit IRC02:55
*** cfarquhar has joined #openstack-infra02:55
*** baoli has joined #openstack-infra02:59
*** dimtruck is now known as zz_dimtruck03:00
*** IlyaG has joined #openstack-infra03:00
*** baoli has quit IRC03:02
*** thorst has joined #openstack-infra03:03
*** IlyaG has quit IRC03:05
*** thorst_ has joined #openstack-infra03:06
*** thorst has quit IRC03:09
*** thorst_ has quit IRC03:11
*** baoli has joined #openstack-infra03:11
*** Daisy has quit IRC03:13
*** Daisy has joined #openstack-infra03:14
*** amitgandhinz has joined #openstack-infra03:15
*** mhayden has joined #openstack-infra03:16
*** Daisy has quit IRC03:18
*** zz_dimtruck is now known as dimtruck03:25
*** IlyaG has joined #openstack-infra03:25
*** dimtruck is now known as zz_dimtruck03:28
*** zz_dimtruck is now known as dimtruck03:28
*** amitgandhinz has quit IRC03:28
*** links has joined #openstack-infra03:33
*** dimtruck is now known as zz_dimtruck03:38
*** achanda has quit IRC03:44
*** ramishra has quit IRC03:47
*** IlyaG has quit IRC03:48
*** piet has joined #openstack-infra03:51
*** ramishra has joined #openstack-infra03:51
*** yuanying has quit IRC03:53
*** baoli has quit IRC03:58
*** zz_dimtruck is now known as dimtruck03:59
*** Daisy has joined #openstack-infra04:00
*** asettle has joined #openstack-infra04:02
*** Jeffrey4l has joined #openstack-infra04:02
*** Daisy has quit IRC04:04
*** Nikolay_St has joined #openstack-infra04:04
*** salv-orlando has joined #openstack-infra04:06
*** salv-orlando has quit IRC04:06
*** piet has quit IRC04:07
*** salv-orlando has joined #openstack-infra04:07
*** salv-orl_ has quit IRC04:09
*** thorst has joined #openstack-infra04:10
*** salv-orlando has quit IRC04:12
*** thorst has quit IRC04:16
openstackgerritwatanabe isao proposed openstack-infra/devstack-gate: Add a fork for Ironic vender driver
*** piet has joined #openstack-infra04:21
*** amitgandhinz has joined #openstack-infra04:25
*** vikram has joined #openstack-infra04:31
vikramI want to add 'onos' driver in marketplace04:31
vikramWhat I am suppose to do for this?04:31
openstackgerritAnusha Ramineni proposed openstack-infra/project-config: Enable Murano service as part of congress job
vikramfungi: ping04:34
vikramfungi: We want to add 'ONOS' driver support in "".. What I am suppose to do for that?04:35
*** amitgandhinz has quit IRC04:38
*** Sukhdev has joined #openstack-infra04:43
*** Daisy has joined #openstack-infra04:43
*** Daisy has quit IRC04:44
*** Daisy has joined #openstack-infra04:45
*** Daisy has quit IRC04:46
*** Daisy has joined #openstack-infra04:47
openstackgerritMasayuki Igawa proposed openstack/os-testr: Add pypi download + version badges into README.rst
*** yuanying has joined #openstack-infra04:56
*** vikram has quit IRC05:02
*** vikram_ has joined #openstack-infra05:04
*** Goneri has quit IRC05:06
*** Nikolay_St has quit IRC05:10
*** Daisy has quit IRC05:12
*** thorst has joined #openstack-infra05:12
*** Daisy has joined #openstack-infra05:13
*** sputnik13 has joined #openstack-infra05:17
*** sputnik13 has quit IRC05:17
*** piet has quit IRC05:20
*** thorst has quit IRC05:21
*** asettle has quit IRC05:23
openstackgerritvikram.choudhary proposed openstack-infra/project-config: Adding neutron-bgp as a part of neutron project.
*** dimtruck is now known as zz_dimtruck05:26
*** sridhar_ram has joined #openstack-infra05:29
*** achanda has joined #openstack-infra05:31
*** amitgandhinz has joined #openstack-infra05:35
*** asettle has joined #openstack-infra05:35
*** yuanying has quit IRC05:40
*** asettle has quit IRC05:40
*** fedexo has joined #openstack-infra05:41
*** roxanaghe has joined #openstack-infra05:45
*** roxanaghe has quit IRC05:47
*** abregman has joined #openstack-infra05:47
*** yuanying has joined #openstack-infra05:48
*** HeOS has quit IRC05:48
*** amitgandhinz has quit IRC05:49
openstackgerritfumihiko kakuma proposed openstack/requirements: Delete networking-ofagent from projects.txt
*** vikram_ has quit IRC05:55
*** HeOS has joined #openstack-infra05:56
*** vikram_ has joined #openstack-infra05:59
*** [HeOS] has joined #openstack-infra06:00
*** HeOS has quit IRC06:02
*** baoli has joined #openstack-infra06:10
*** harlowja_at_home has quit IRC06:12
*** sridhar_ram has quit IRC06:12
*** Nikolay_St has joined #openstack-infra06:13
*** baoli has quit IRC06:15
*** zz_dimtruck is now known as dimtruck06:18
*** thorst has joined #openstack-infra06:19
*** [HeOS] has quit IRC06:19
*** salv-orlando has joined #openstack-infra06:21
*** harlowja_at_home has joined #openstack-infra06:23
*** Daisy_ has joined #openstack-infra06:25
*** thorst has quit IRC06:26
*** mylu has quit IRC06:27
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** Daisy has quit IRC06:28
*** Daisy_ has quit IRC06:28
*** HeOS has joined #openstack-infra06:28
*** Daisy has joined #openstack-infra06:28
*** maishsk has joined #openstack-infra06:30
*** salv-orlando has quit IRC06:34
*** [HeOS] has joined #openstack-infra06:39
*** HeOS has quit IRC06:41
*** fedexo has quit IRC06:41
*** amitgandhinz has joined #openstack-infra06:45
*** oanson has joined #openstack-infra06:47
*** roxanaghe has joined #openstack-infra06:47
*** [HeOS] has quit IRC06:48
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** pcaruana has joined #openstack-infra06:49
*** nadya has joined #openstack-infra06:51
*** roxanaghe has quit IRC06:52
*** HeOS has joined #openstack-infra06:56
*** nadya has quit IRC06:57
*** amitgandhinz has quit IRC06:58
*** Liuqing has joined #openstack-infra06:59
*** Daisy_ has joined #openstack-infra07:02
*** Daisy has quit IRC07:02
*** HeOS has quit IRC07:06
*** Daisy has joined #openstack-infra07:07
*** Daisy_ has quit IRC07:07
*** HeOS has joined #openstack-infra07:08
*** vikram_ has quit IRC07:09
*** vikram_ has joined #openstack-infra07:09
*** e0ne has joined #openstack-infra07:15
*** e0ne has quit IRC07:18
*** asettle has joined #openstack-infra07:24
*** thorst has joined #openstack-infra07:24
*** asettle has quit IRC07:29
*** thorst has quit IRC07:31
*** nmagnezi has joined #openstack-infra07:34
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
openstackgerritAndrey Nikitin proposed openstack-infra/project-config: 'fuel-abandon' group added to Fuel-related projects
*** binbincong has quit IRC07:47
*** amitgandhinz has joined #openstack-infra07:55
*** |-paul-| has quit IRC07:56
*** tqtran has joined #openstack-infra07:58
*** sabari- is now known as sabari08:01
*** salv-orlando has joined #openstack-infra08:01
*** binbincong has joined #openstack-infra08:04
*** salv-orlando has quit IRC08:07
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: midonet: Make tempest jobs voting
*** amitgandhinz has quit IRC08:08
*** nadya has joined #openstack-infra08:13
*** e0ne has joined #openstack-infra08:14
*** nihilifer has quit IRC08:16
*** nihilifer has joined #openstack-infra08:17
*** hichihara has quit IRC08:19
*** thorst has joined #openstack-infra08:30
*** yuanying has quit IRC08:31
*** yuanying has joined #openstack-infra08:31
*** GheRiver1 has joined #openstack-infra08:32
*** yuanying has quit IRC08:32
*** salv-orlando has joined #openstack-infra08:33
*** roxanaghe has joined #openstack-infra08:36
*** thorst has quit IRC08:36
*** ociuhandu has quit IRC08:36
*** dizquierdo has joined #openstack-infra08:36
*** salv-orlando has quit IRC08:37
*** yuanying has joined #openstack-infra08:39
*** roxanaghe has quit IRC08:40
*** armax has joined #openstack-infra08:44
*** armax has quit IRC08:49
*** mikelk has joined #openstack-infra08:53
*** mikelk has quit IRC08:55
*** jcooley_ has quit IRC08:56
*** GheRivero has quit IRC08:57
*** Sukhdev has quit IRC08:59
*** GheRivero has joined #openstack-infra09:01
*** GheRiver1 has quit IRC09:03
*** achanda has quit IRC09:04
*** amitgandhinz has joined #openstack-infra09:05
*** thorst has joined #openstack-infra09:08
*** lezbar has joined #openstack-infra09:09
*** ociuhandu has joined #openstack-infra09:10
*** achanda has joined #openstack-infra09:11
*** thorst has quit IRC09:15
*** yuanying has quit IRC09:16
*** Qiming has quit IRC09:17
*** Qiming has joined #openstack-infra09:17
*** amitgandhinz has quit IRC09:18
*** skolekonov|away is now known as skolekonov09:19
*** achanda has quit IRC09:28
*** kzaitsev_mb has joined #openstack-infra09:31
*** thorst has joined #openstack-infra09:32
*** salv-orlando has joined #openstack-infra09:36
*** thorst has quit IRC09:37
*** yamahata has quit IRC09:37
*** pcaruana has quit IRC09:42
*** GheRiver1 has joined #openstack-infra09:43
*** zeih has joined #openstack-infra09:43
*** Daisy has quit IRC09:44
*** Daisy has joined #openstack-infra09:44
*** salv-orlando has quit IRC09:46
*** Daisy has quit IRC09:46
*** Daisy has joined #openstack-infra09:47
*** lezbar has quit IRC09:48
*** Daisy has quit IRC09:48
*** vgridnev has joined #openstack-infra09:53
*** pcaruana has joined #openstack-infra09:55
*** dizquierdo has quit IRC10:01
*** maishsk has quit IRC10:08
*** maishsk has joined #openstack-infra10:10
*** amitgandhinz has joined #openstack-infra10:15
openstackgerritMerged openstack-infra/project-config: 'fuel-abandon' group added to Fuel-related projects
*** aloga_ is now known as aloga10:18
*** akaszuba has joined #openstack-infra10:18
*** Qiming has quit IRC10:24
*** roxanaghe has joined #openstack-infra10:24
*** akaszuba has quit IRC10:24
*** Liuqing has quit IRC10:25
*** sdague has joined #openstack-infra10:27
*** roxanaghe has quit IRC10:28
*** amitgandhinz has quit IRC10:28
*** achanda has joined #openstack-infra10:28
*** tqtran has quit IRC10:30
*** achanda has quit IRC10:33
*** thorst has joined #openstack-infra10:36
*** achanda has joined #openstack-infra10:36
*** vikram_ has quit IRC10:39
*** thorst has quit IRC10:42
*** GheRiver1 has quit IRC10:43
*** |-paul-| has joined #openstack-infra10:46
*** Liuqing has joined #openstack-infra10:51
*** asettle has joined #openstack-infra10:56
*** nadya has quit IRC10:56
*** maishsk has quit IRC10:56
*** yamamoto has quit IRC10:59
*** dizquierdo has joined #openstack-infra11:04
*** pcaruana has quit IRC11:09
*** zeih has quit IRC11:10
*** salv-orlando has joined #openstack-infra11:11
*** achanda has quit IRC11:11
*** salv-orlando has quit IRC11:13
*** Liuqing_ has joined #openstack-infra11:15
*** Liuqing has quit IRC11:16
*** maishsk has joined #openstack-infra11:17
*** tpsilva has joined #openstack-infra11:18
*** ldnunes has joined #openstack-infra11:18
*** pcaruana has joined #openstack-infra11:23
*** nadya has joined #openstack-infra11:24
*** amitgandhinz has joined #openstack-infra11:25
*** thorst has joined #openstack-infra11:26
*** tqtran has joined #openstack-infra11:26
*** doug-fish has joined #openstack-infra11:27
*** jpr has joined #openstack-infra11:29
*** nadya has quit IRC11:29
*** Qiming has joined #openstack-infra11:30
*** tqtran has quit IRC11:31
*** GheRiver1 has joined #openstack-infra11:33
*** gordc has joined #openstack-infra11:33
*** yfried has joined #openstack-infra11:35
*** vgridnev has quit IRC11:38
*** amitgandhinz has quit IRC11:38
*** maishsk has quit IRC11:41
*** nadya has joined #openstack-infra11:41
*** |-paul-| has quit IRC11:43
*** flaper87 has quit IRC11:48
*** wfoster has quit IRC11:49
HeOSHello, team! I have merged my request: So, could you add me to the group, to have ability to add a bot-user into the group?11:50
*** wfoster has joined #openstack-infra11:50
*** flaper87 has joined #openstack-infra11:51
*** maishsk has joined #openstack-infra11:55
*** weshay has joined #openstack-infra11:56
*** baoli has joined #openstack-infra11:57
*** maishsk has quit IRC11:59
*** Liuqing_ has quit IRC12:02
*** asettle has quit IRC12:05
*** zeih has joined #openstack-infra12:09
*** roxanaghe has joined #openstack-infra12:12
*** maishsk has joined #openstack-infra12:12
*** jaypipes has joined #openstack-infra12:12
*** achanda has joined #openstack-infra12:12
*** aysyd has joined #openstack-infra12:13
*** zeih has quit IRC12:16
*** roxanaghe has quit IRC12:17
*** achanda has quit IRC12:17
*** vgridnev has joined #openstack-infra12:23
*** rlandy has joined #openstack-infra12:23
*** dprince has joined #openstack-infra12:24
*** ZZelle has joined #openstack-infra12:24
*** ilyashakhat has joined #openstack-infra12:26
*** erlon has joined #openstack-infra12:27
AJaegerproject-config cores, there's a patch to add a --verbose option to It has seen little review. Is that something we want?
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI
*** yamamoto has joined #openstack-infra12:28
*** Daisy has joined #openstack-infra12:29
*** edmondsw has joined #openstack-infra12:30
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with non-HA jobs
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: midonet: Make tempest jobs voting
*** _degorenko is now known as degorenko12:32
AJaegerinfra-root, oslo.service is in the release since 9 hours now - waiting for wheel-release-cent7x64 to build . Are we missing a node again?12:32
*** amrith is now known as _amrith_12:33
*** Daisy has quit IRC12:33
*** amitgandhinz has joined #openstack-infra12:34
ilyashakhatSergeyLukjanov: please do me a favor and add me to performa-core and performa-release groups :)12:35
*** julim has joined #openstack-infra12:36
*** zhurong has joined #openstack-infra12:37
*** pcaruana has quit IRC12:39
*** kgiusti has joined #openstack-infra12:40
*** ilyashakhat has quit IRC12:42
*** yamamoto has quit IRC12:44
*** yamamoto has joined #openstack-infra12:47
*** piet has joined #openstack-infra12:47
*** amitgandhinz has quit IRC12:48
*** amitgandhinz has joined #openstack-infra12:49
*** salv-orlando has joined #openstack-infra12:50
sdague  is quite surprising12:51
*** amitgandhinz has quit IRC12:51
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again
*** yamamoto has quit IRC12:52
*** pcaruana has joined #openstack-infra12:52
*** salv-orlando has quit IRC12:53
*** amitgandhinz has joined #openstack-infra12:54
*** Nikolay_St has quit IRC12:55
*** yamamoto has joined #openstack-infra12:55
*** amitgandhinz has quit IRC12:56
*** kushal has quit IRC12:56
*** annegentle has joined #openstack-infra12:56
*** amitgandhinz has joined #openstack-infra12:56
*** dimtruck is now known as zz_dimtruck13:01
*** huzhj_ has joined #openstack-infra13:01
*** Nikolay_St has joined #openstack-infra13:02
*** abregman has quit IRC13:02
pabelangerAJaeger: Not missing, but it is about ~8mins slower then our ubuntu-trusty wheel mirrors. So it's been getting backed up in the pipeline13:04
pabelangerAJaeger: is the node13:05
*** jcoufal has joined #openstack-infra13:05
*** yfried has quit IRC13:06
pabelangerUbuntu trusty wheel is ~40mins where Centos 7 wheel is 48mins13:06
pabelanger vs
*** yamamoto has quit IRC13:12
*** mtanino has quit IRC13:13
*** achanda has joined #openstack-infra13:15
*** rcrit_ has quit IRC13:16
*** julim has quit IRC13:17
*** yamamoto has joined #openstack-infra13:17
*** abregman has joined #openstack-infra13:17
*** _ari_ has joined #openstack-infra13:18
annegentleAJaeger: you know about checksum failures? Just saw this it on this patch
*** kushal has joined #openstack-infra13:19
*** achanda has quit IRC13:19
*** maishsk has quit IRC13:20
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove sbcl from bindep-fallback.txt
fungiAJaeger: the wheel release jobs run from the wheel-build workers, which are only able to run one job at a time13:20
*** mtanino has joined #openstack-infra13:21
*** yamamoto has quit IRC13:21
*** maishsk has joined #openstack-infra13:21
*** ayoung has joined #openstack-infra13:22
*** annegentle has quit IRC13:23
*** julim has joined #openstack-infra13:23
*** zz_dimtruck is now known as dimtruck13:24
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add distro package install for run-tests
AJaegerfungi, it runs very slow. We shouldn't have to wait 9hours to release 10 packages or so...13:25
*** annegentle has joined #openstack-infra13:25
*** annegentle has quit IRC13:25
fungiAJaeger: well, all the important release jobs completed. those are just wheel mirror updates (and very redundant ones at this point)13:25
*** jsavak has joined #openstack-infra13:25
AJaegerpabelanger: the centos7 wheel looks far slower than those 8 mins. To me it sounds more like 80 mins instead of 48 ;(13:25
fungithe pypi upload completed and announcement was sent for oslo.utils >10 hours ago13:26
AJaegerfungi: yes, indeed. Just still seeing now three jobs there - plus the one in the periodic queue, is odd.13:26
fungithe one in post looks like a stuck job. it's been running for over 5 hours13:27
*** mtanino has quit IRC13:27
*** rfolco_ has joined #openstack-infra13:27
AJaegerannegentle: That's again a network problem on rax-ord ;(13:27
*** tqtran has joined #openstack-infra13:28
*** piet has quit IRC13:28
fungilooks like it stuck trying to upload logs to swift almost 5 hours ago13:28
AJaegerfungi, we still have network problems on rax-ord. Not sure whether there's anything we can do ;(13:28
fungiespecially annoying since rackspace moved most of our quota to ord13:28
AJaegerfungi: ;(13:28
* AJaeger will recheck annegentle's job...13:29
fungithe increase in jobs running there may explain the increase in network related issues13:29
fungioh, annegentle also had a rax-ord network issue on a job? i thought you were talking about the stuck swift upload for that heat coverage job (which is also in rax-ord)13:30
AJaegerfungi, I'm currently trying to move sbcl out of bindep-fallback and run into 298230, could you check that one later, please?13:30
AJaegerfungi: I was looking at backscroll - didn't look at post ;)13:30
*** tqtran has quit IRC13:32
AJaegerthanks, fungi13:32
fungiyou're welcome!13:33
AJaegerpabelanger, fungi: Trying to get packages out of bindep-fallback is larger untertaking for some of them. firefox is used by 16+ repositories...13:34
*** mriedem has joined #openstack-infra13:35
fungiand may be hard to find them all13:35
fungiwithout some collateral damage anyway13:35
AJaegerand will need backports of other-requirements ;(13:35
*** abregman has quit IRC13:35
*** links has quit IRC13:36
*** piet has joined #openstack-infra13:36
openstackgerritMerged openstack-infra/project-config: Adding Mellanox Extension for the OpenStack Dashboard (Horizon)
pabelangerAJaeger: Ya, I imagine it will take some trial and error13:38
openstackgerritMerged openstack-infra/project-config: Adding searchlight-ui repo and jobs
* AJaeger will be back later13:39
*** akshai has joined #openstack-infra13:45
*** nmagnezi has quit IRC13:46
*** yamamoto has joined #openstack-infra13:47
openstackgerritMerged openstack-infra/project-config: Add distro package install for run-tests
*** armax has joined #openstack-infra13:48
*** openstackgerrit has quit IRC13:48
*** abregman has joined #openstack-infra13:48
*** openstackgerrit has joined #openstack-infra13:48
*** regXboi has joined #openstack-infra13:49
openstackgerritMerged openstack-infra/project-config: Run the v3 only job in neutron
*** dimtruck is now known as zz_dimtruck13:50
AJaegerfungi, pabelanger: We have now bindep-fallback both as list of packages to install - and as list of packages to cache. If we remove entries from it and move them to other-requirements, should we create a cache file in addition?13:51
*** armax has quit IRC13:52
*** yamamoto has quit IRC13:52
AJaegerthat cache file would include packages that some projects install  via other-requirements but are not installed by default.13:52
openstackgerritMerged openstack-infra/project-config: Make monasca persister tempest tests voting
*** annegent_ has joined #openstack-infra13:52
*** annegent_ has quit IRC13:52
*** annegent_ has joined #openstack-infra13:53
AJaegerfungi: wow! Didn't know that we do it already. thanks13:55
*** _amrith_ is now known as amrith13:56
*** nadya has quit IRC13:56
*** eharney has joined #openstack-infra13:56
pabelangerYa, we do use other-requirements.txt as a cache file for DIBs.  But the issue is, we only validate other-requirements.txt ubuntu-trusty. I've been wrapping my brain around how to test it over other OSes13:56
openstackgerritTom Cocozzello proposed openstack-infra/project-config: Experimental patch_cover gate job
AJaegerpabelanger: thanks for implementing that (if the NOTE is correct)13:56
*** nadya has joined #openstack-infra13:56
AJaegerfungi, did we have a bindep release? See
*** nadya has quit IRC13:56
fungipabelanger: well... we validate it for whatever jobs a given project runs. but yes it's probably quite easy for a project not running any install-distro-packages jobs on fedora to include an entry which breaks package caching on fedora. i think it was suggested there may be an option to tell the package manager to ignore missing packages?13:58
AJaegerpabelanger: you could run a periodic test that goes through all files and installs on fedora etc.13:58
*** sigmavirus24_awa is now known as sigmavirus2413:59
pabelangerRight, I was thinking of a periodic job for other-requirements for each other OS then ubuntu-trusty13:59
fungiAJaeger: merged as 4db1a7a and `git tag --contains 4db1a7a` says 0.1.0 so we can update that element safely now, i think13:59
fungithanks for spotting13:59
*** roxanaghe has joined #openstack-infra14:00
fungiyep, just checked out 0.1.0 specifically and the support for using "-f -" to read stdin is included14:01
AJaegerfungi: great. Will you update?14:01
fungiworking on it now14:01
*** bryan_att has joined #openstack-infra14:02
*** knikolla has joined #openstack-infra14:04
*** roxanaghe has quit IRC14:04
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Drop tempfile workaround for aggregate bindep list
fungiAJaeger: ^14:05
HeOSInfra-team, I want to repeat my question: what should I do to add myself and Fuel DevOps Robot to the following group,members? :)14:06
pabelangerAJaeger: we could do something like 297809 but for other-requirements.txt, and allow projects to add it as a template14:06
*** rbrndt_ has joined #openstack-infra14:08
AJaegerNeat idea - Just install all packages when other-requirements is updated.14:08
fungithat means making sure every project which might create an other-requirements.txt file (which could be _any_ project) has those jobs configured14:08
AJaegerwe could add this to python-jobs, so that every project has it ;)14:08
AJaegerAnd then add a template similar to check-requirements14:09
fungithere are tons of projects which don't use the python-jobs group14:09
pabelangerwell, to start, we could just have people add the template themselves if they are interested14:10
AJaegerfungi: you're right, many use it. But adding a new job-group would be nice14:10
pabelangeruntil we weed out the bugs14:10
fungii think instead we should make the caching robust so that an untested but broken other-requirements.txt file doesn't stop image builds/package caching14:10
AJaegerpabelanger: I'm working on ha-guide setup right now - as docs core - so, I volunteer to guinea pig that repo ;)14:10
pabelangerAJaeger: Ya, odyssey4me has all the openstack ansible projects setup too14:11
fungialso the nova example is up for review now at
fungimriedem has already provided awesome feedback ;)14:11
pabelangerAJaeger: let me see about setting up a template14:11
pabelangerfungi: Ooohh, nice14:11
*** yamamoto has joined #openstack-infra14:12
AJaegerfungi, I think we need both - the repo job will help projects spot problems but the robust caching is essential...14:13
*** zeih has joined #openstack-infra14:13
Shrewsmtreinish: if I +A 285615, can you add a follow up with a new reno release note?14:14
Shrewsmtreinish: or you can add it to the current review14:15
mtreinishShrews: sure I can do that14:15
mtreinishI'm all for stat padding14:15
fungiAJaeger: well, the ideal way to validate the package list is to run useful jobs for those platforms, more than just testing installation of the package list (and i can see projects which explicitly don't care about fedora 23 support not wanting a voting job which tests that their package list installation doesn't break on fedora 23)14:15
Shrewsmtreinish: approved14:16
AJaegerfungi: yeah, we have many projects only caring about one OS ;(14:16
*** annegent_ has quit IRC14:18
*** zeih has quit IRC14:18
AJaegerfungi, btw. do you want to kill the heat post job? Or what should be done with it?14:18
dmsimardIt looks like Jenkins 2 release candidate is meant so ship sometime in April. Have you checked it out ? Any incentive to upgrade ?14:18
fungiAJaeger: i'll kill it14:19
pabelangerdmsimard: likely not. I suspect we want to land zuulv3 first14:19
*** doug-fis_ has joined #openstack-infra14:19
Shrewsmtreinish: oh, actually, i'm going to go ahead and approve mordred's follow up, so if your note could reference the new wait_for_image() too, that would be awesome14:19
pabelangerwhich will remove the need for jenkins14:19
dmsimardpabelanger: fair enough, are you getting closer to v3 now ?14:19
dhellmannhas anyone else reported a babel problem "AttributeError: '' object has no attribute 'hour'"?14:19
fungidmsimard: yeah, we're planning to stop using jenkins soonish via
mtreinishShrews: sure, I'll add that too14:20
*** knikolla has quit IRC14:20
fungidmsimard: late in the newton release cycle or maybe early in ocata? hard to say for sure, depends on how many people pitch in as the momentum builds14:20
dmsimardfungi: thanks :D14:21
pabelangerdmsimard: we have a branch up: and ansible is launching jobs. Like fungi says, more people helping out, the faster it will land.14:21
*** knikolla has joined #openstack-infra14:21
*** doug-fish has quit IRC14:21
*** ajmiller has joined #openstack-infra14:22
openstackgerritMatthew Treinish proposed openstack-infra/shade: Add release notes for new create_image_snapshot() args
mtreinishShrews: ^^^14:22
Shrewsmtreinish: thx14:23
dimsnetwork issues - "Download error on [Errno 101] Network is unreachable -- Some packages may not be found!"14:23
dimsreading scrollback, sounds like known issues :)14:24
*** tiswanso has joined #openstack-infra14:24
fungidims: yeah, i'm worried that we may need to stop using rax-ord for a while14:25
fungiis that still going on?14:25
fungii know they postponed some of their disruptive network maintenance to last night or tonight14:25
dhellmannoh, fun, a new version of sphinx seems to be breaking the release repo doc build14:25
*** zz_dimtruck is now known as dimtruck14:27
*** akshai has quit IRC14:29
*** smarcet has joined #openstack-infra14:29
*** akshai has joined #openstack-infra14:29
pabelangerfungi: we never did restart zuul because of the memory leak14:30
*** jamesmcarthur has joined #openstack-infra14:31
*** Hal has joined #openstack-infra14:31
*** Hal is now known as Guest2126314:31
smarcetfungi: morning, seems that openstackid-dev is down14:31
*** annegentle has joined #openstack-infra14:31
fungismarcet: down and you don't know why, or down and you're flooding it with requests?14:32
jamesmcarthurdown and we don’t know why14:32
smarcetfungi: down and dunno14:32
smarcetseems DNS problem14:32
jamesmcarthurping openstackid-dev.org14:33
jamesmcarthurping: cannot resolve Unknown host14:33
*** Nikolay_St has quit IRC14:33
fungithat has never existed14:34
fungiare you sure you're not thinking of
*** jpr has quit IRC14:34
fungilooks like it's up for me(and that's the name it's always had)14:34
jamesmcarthurok - web team going for coffee14:35
fungiheh. i could use some more myself ;)14:35
* fungi imagines Clint walking like an egyptian14:36
* morgan is a coffee zombie still.14:36
Clintall the better for you to walk like an egyptian in my stead14:37
fungipabelanger: yep, looks like we edged into swap over the weekend
fungipabelanger: we should probably coordinate a restart with the release team so that we don't disrupt any ongoing release activities by dropping jobs in ancillary pipelines14:38
*** markvoelker has joined #openstack-infra14:38
*** jamesmcarthur has quit IRC14:38
jeblairwe might also experimentally try restarting it without the foreign pipeline patch, or without connections.14:38
*** jamesmcarthur has joined #openstack-infra14:38
jeblairto narrow down which of those is the cause of the leak14:39
fungii'm in favor as long as one of they can still be easily untangled14:39
fungier, as long as they can still be easily untangled14:40
fungiwe're not relying on either of those features right?14:40
fungithe foreign pipelines change is really for third parties using a subset of our projects14:41
fungiand connections is a stepping stone to the github trigger and such?14:41
jeblairyep.  though i should have called it 'foreign project'.14:41
fungieither (or both) seem safe to me14:42
jeblair(though mostly originally written so we could use two accounts for the same gerrit)14:42
*** sdake has joined #openstack-infra14:42
*** denisra has joined #openstack-infra14:42
*** mylu has joined #openstack-infra14:42
dimsjeblair fungi : hmm, ec2-api went awol on gerrit -
*** yfried has joined #openstack-infra14:43
oansonI got 'ValueError: '/usr/local/lib/python2.7/dist-packages' is not in list' in our CI tests using virtualenv 14.0.6. Forcing the gate to use 15.0.1 solved the issue.14:43
*** nelsnelson has joined #openstack-infra14:43
fungiremoving both would allow us to disprove the theory that the leak is caused by one of them, i guess, but picking one to remove seems marginally more likely to yield a quicker answer since chances are it is one of them14:43
oansonAm I right in understanding that this sort of change would require upper-constraints to be raised for virtualenv?14:43
openstackgerritMerged openstack-infra/project-config: Drop tempfile workaround for aggregate bindep list
*** mylu has quit IRC14:44
*** asselin has joined #openstack-infra14:44
fungioanson: we don't constrain virtualenv, we constrain the things we install in a virtualenv. there are bootstrapping issues with managing pip, setuptools, virtualenv and tox in global requirements14:44
*** mylu has joined #openstack-infra14:44
oansonfungi: So the version of virtualenv on the gate is not decided by upper-constraints?14:45
*** jamesmcarthur has quit IRC14:46
*** sdake has quit IRC14:46
fungioanson: depends on the job (i think if you're running under devstack, it may use that to decide what version of virtualenv to use when setting up tempest)14:46
*** sdake has joined #openstack-infra14:47
fungioanson: most of our jobs don't use devstack, and run with whatever version of virtualenv we preinstall on our servers14:47
oansonfungi: I see. I think this job does run devstack, but I will verify.14:48
fungioanson: since you don't say what job you're encountering this with, i don't really know14:48
oansonThanks for your help.14:48
oansonIt's on dragonflow fullstack tests14:48
fungivirtualenv in constraints with devstack is sort of a corner case for us14:48
*** annegentle has quit IRC14:48
*** Daisy has joined #openstack-infra14:49
fungidims: oh, wow, that is all sorts of broken. the openstack/ec2-api repo is still present on our git servers but gerrit is pitching permission errors if i try to view any changes linked from it14:50
oansonfungi, thanks again for your help.14:50
dimsfungi : guess we need to see who else is awol14:50
openstackgerritMerged openstack-infra/shade: Add wait support to create_image_snapshot()
fungidims: well, i'm looking for logs now. it's possible this is fallout from network disruption to cinder or trove14:51
dimsfungi : thanks14:51
fungidoesn't seem to be filesystem related (nothing screaming recently in dmesg)14:52
*** burgerk has joined #openstack-infra14:53
*** annegentle has joined #openstack-infra14:53
*** Daisy has quit IRC14:54
openstackgerritMerged openstack-infra/shade: Split waiting for images into its own method
openstackgerritMerged openstack-infra/shade: Fix grant_role docstring
fungiinfra-root: heads up ^ we may have gerrit corruption of some sort underway14:54
fungi[2016-03-28 14:54:44,073] WARN : Cannot read project openstack/ec2-api14:55
*** mylu has quit IRC14:56
*** asselin has quit IRC14:56
*** mylu has joined #openstack-infra14:56
fungihere's the first thing i can find14:56
fungi[2016-03-27 05:01:15,884] WARN : Pack file /home/gerrit2/review_site/git/openstack/ec2-driver.git/objects/pack/pack-193d82faea4408f0191a06cde2c253abb493fead.pack was deleted, removing it from pack list14:56
fungilots of that for a variety of projects, so might just be normal git pack related messages14:57
fungihowever these two are adjacent14:57
fungi[2016-03-27 05:41:19,448] WARN : Pack file /home/gerrit2/review_site/git/openstack/ec2-api.git/objects/pack/pack-3a686787179a5e70d4c53e9be6897ec5ed0821b2.pack was deleted, removing it from pack list14:57
fungi[2016-03-27 05:41:19,450] WARN : Cannot read project openstack/ec2-api14:57
jeblairthe first message makes me wonder if we should flush the gerrit cache14:57
*** fawadkhaliq has joined #openstack-infra14:58
*** maishsk has quit IRC14:58
*** asselin has joined #openstack-infra14:58
fungigrepping for "Cannot read project" it seems to be only openstack/ec2-api affected luckily14:58
*** annegentle has quit IRC14:59
*** ilyashakhat has joined #openstack-infra14:59
fungii can call flush-caches15:00
*** zhurong has quit IRC15:00
*** jsavak has quit IRC15:00
fungiunless there's a good reason to worry it will make matters worse15:00
*** mylu has quit IRC15:00
jeblairfungi: note your first errer message was for ec2-driver, not ec2-api15:00
fungiaha, yep15:00
fungibut the adjacent packfile removal and read error were for ec2-api15:01
*** jsavak has joined #openstack-infra15:01
jeblairfungi: i copied the repo and ran git fsck --full; nothing alarming (only some dangling blobs which are expected due to our lack of gc)15:01
fungizaro: any reason to worry about calling flush-caches in a situation like this, do you think?15:02
*** sputnik13 has joined #openstack-infra15:03
*** mylu has joined #openstack-infra15:03
*** LaT0rtue_ has joined #openstack-infra15:04
fungidims: i take it you discovered the query issue with ec2-api while trying to do semi-urgent release work?15:06
fungii can go ahead and give teh cache flush a try, and next on the list of possible terrible solutions is probably a gerrit service restart15:06
dimsfungi : this was raised on mailing list15:06
fungiahh, busy morning so haven't gotten through e-mail from overnight15:07
dimsfungi : not urgent from release perspective, just concerned if we should tag releases or not15:07
*** edmondsw has quit IRC15:07
fungicache flush is running now, will keep an eye out for new and exciting errors15:07
dhellmanndims : I'm going to make the call that we wait to tag anything until this is sorted. I don't want to have to retag and risk having irreproducible deliverables.15:08
fungii don't like flush-caches because it makes gerrit pretty unresponsive for several minutes15:08
dimsdhellmann : +115:08
*** jamesmcarthur has joined #openstack-infra15:08
fungidhellmann: agreed, would like to make sure this is just a cosmetic issue with our git repack and gerrit's persistent cache fighting, rather than the start of something more significant15:09
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual: Document other-requirements
AJaegerfungi, pabelanger, could you review this later, please? ^15:10
jeblairfungi: i don't necessarily think the repack is related.  it could be, but i don't know we have a strong correlation yet.15:10
dhellmannfungi : yep. let me know if I can help, otherwise I'll monitor the discussion here15:11
fungijeblair: right, we have a sample size of 1. i'm not confident we'll know what's caused it without seeing it happen more often15:11
*** annegentle has joined #openstack-infra15:11
*** jcoufal has quit IRC15:12
fungicache flush doesn't seem to have solved this15:12
*** esker has quit IRC15:12
*** esker has joined #openstack-infra15:13
jeblairfungi: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 72803eb8013ff25d7751b4a7c6cd6571d663143515:14
*** jamesmcarthur has quit IRC15:14
jeblairfungi: i can see that object15:14
*** oanson has quit IRC15:14
*** jamesmcarthur has joined #openstack-infra15:14
fungiyep. this seems like the git operation timeout issue we saw back when we tried upgrading to... 2.9?15:14
*** akshai has quit IRC15:15
*** akshai has joined #openstack-infra15:15
jeblair(it's no wonder it's not displaying at all if it thinks it can't read its config history)15:16
fungii can see it in ~gerrit2/review_site/git/openstack/ec2-api.git too15:16
*** yamamoto has quit IRC15:16
jeblairfungi: agreed.  i think we found that only restarting corrected that.15:16
*** yamamoto has joined #openstack-infra15:17
fungiso if there is a repack correlation, it might be that we pounded the fs with i/o or possibly hung for a bit updating that object (causing reads to block) right when gerrit tried to read it15:17
*** vgridnev has quit IRC15:17
*** achanda has joined #openstack-infra15:17
fungiokay, any objections to an emergency gerrit restart?15:17
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual: Document other-requirements
jeblairfungi: i thought the 'timeout -> failed read' bug was corrected15:17
jeblairfungi: no objections to restart.15:17
*** dizquierdo has quit IRC15:17
*** cloudtrainme has joined #openstack-infra15:18
fungii'll get the full traceback into a paste once the restart is going15:18
*** vgridnev has joined #openstack-infra15:18
*** sbalukoff has quit IRC15:18
anteayafungi: no objection to restart15:19
*** woodster_ has joined #openstack-infra15:19
fungi#status notice Gerrit is restarting on in an attempt to address an issue reading an object from the ec2-api repository15:20
openstackstatusfungi: sending notice15:20
fungilooks like it should be back up now15:21
-openstackstatus- NOTICE: Gerrit is restarting on in an attempt to address an issue reading an object from the ec2-api repository15:21
*** yamamoto has quit IRC15:21
fungilooks like that solved it15:22
*** rraja has joined #openstack-infra15:22
fungiet cetera15:22
dansmithfungi: have you seen some extra failures talking to this morning in the tests?15:22
openstackstatusfungi: finished sending notice15:22
dansmithI thought that you were doing a restart for that reason, but I see not15:22
fungidansmith: we've seen some, all of which seem to be for workers in rax-ord15:22
*** achanda has quit IRC15:22
dansmithyeah, in rax-ord15:23
fungiit seems like rackspace is having some sort of network issues not reported on
dansmiththey were last week too right?15:24
*** dmellado is now known as dmellado|off15:24
fungithey also shifted most of our quota to ord due to load problems in iad15:24
fungiand the uptick in network problems we're seeing for jobs in ord seems like it might roughly correspond15:25
openstackgerritBen Nemec proposed openstack-infra/project-config: Add stable branch jobs for tripleo-ci
dansmithfungi: seems like ever since alaski left, they've had nothing but problems.. coincidence?15:25
*** bpokorny has joined #openstack-infra15:26
fungidims: dhellmann: openstack/ec2-api seems fine in gerrit now since the restart15:26
alaskidansmith: hah15:26
dimsfungi : cool thanks!15:26
fungidansmith: alaski: correlation implying causation there? ;)15:27
*** armax has joined #openstack-infra15:27
sdaguewow - the git server failure rate is crazy -
dhellmannfungi : thanks!15:27
dansmithfungi: I dunno.. seems clear to me :D15:27
fungisdague: yeah, i'm considering proposing turning off rax-ord in nodepool, though we need someone to write up a coherent trouble ticket15:28
*** yamamoto has joined #openstack-infra15:28
*** yamamoto has quit IRC15:28
*** ddieterly has joined #openstack-infra15:28
*** annegent_ has joined #openstack-infra15:28
sdaguefungi: yeh, it looks almost entirely there15:28
sdaguefungi: how many nodes do we lose with that going off?15:28
fungialmost a couple hundred15:28
*** fawadkhaliq has quit IRC15:28
fungii think15:28
fungisince rackspace asked us to move most of our load to ord15:29
*** Qiming has quit IRC15:29
sdagueany idea if they realize ord is this bad?15:29
fungi195 says nodepool.yaml15:29
*** ilyashakhat has quit IRC15:30
fungisdague: i'm thinking they don't have any clue since makes no mention of it15:30
fungiwhich is why someone should throw together a trouble ticket15:30
fungiinfra-root have access to the rackspace dashboard15:30
*** roxanaghe has joined #openstack-infra15:31
anteayafungi: do you want me to offer the project-config patch to turn off ord?15:31
*** rraja has quit IRC15:31
fungiwe likely need to put together a list of nova instance uuids which exhibited the issue15:31
*** annegentle has quit IRC15:31
*** salv-orlando has joined #openstack-infra15:32
fungii guess this is a holiday/hangoverday for enough people that we're not driving full quota so it's probably okay to take ord out for the moment15:32
sdagueyeh, eu is basically entirely off today15:32
*** tiswanso has quit IRC15:32
elmikoAJaeger: ping15:32
*** tiswanso has joined #openstack-infra15:32
openstackgerritAnita Kuno proposed openstack-infra/project-config: Turn off rax-ord
anteayafungi: done ^^15:33
jeblairif someone can get me a list of some nodes that have exhibited the failure, i will put together the list of uuids and open the trouble ticket15:34
fungithanks jeblair!15:34
fungii'm looking to see how to easily extract that from kibana15:34
*** roxanaghe has quit IRC15:35
openstackgerritAnita Kuno proposed openstack-infra/project-config: Turn off rax-ord
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion with net-iso in periodic job
anteayaI had an errant whitespace15:36
*** Nikolay_St has joined #openstack-infra15:37
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test
fungijeblair: tab-delimited list of times and node names
*** kzaitsev_mb has quit IRC15:39
fungiwill probably want to dedupe the node names15:39
fungithose were the 100 most recent hits15:39
anteayafungi: sorry you had +2'd patchset 1 which had an extra whitespace, which I removed in patchset 215:39
*** keedya has joined #openstack-infra15:39
jeblairfungi: thx15:39
*** mylu has quit IRC15:40
jeblairfungi: those times are when the network error occured?15:40
zxiirodoes git-review support signed pushes?15:40
fungijeblair: yes, according to the logs where those timestamps were obtained15:41
*** salv-orlando has quit IRC15:41
*** torandu has quit IRC15:41
*** torandu has joined #openstack-infra15:41
fungizxiiro: git-review supports git and doesn't break signed commits as far as i've noticed15:41
anteayaI'm going back to Easter Mondaying15:41
AJaegerelmiko: what's up?15:41
*** tphummel has joined #openstack-infra15:42
fungizxiiro: it doesn't explicitly _support_ signed commits, it just treats them like any other commit15:42
zxiirofungi: So latest Gerrit supports "signed pushes" which I think just means you do "git push --sign" or something. I guess git-review needs to be updated to support that flag since I can't find a way to enable ti.15:42
*** roxanaghe has joined #openstack-infra15:42
fungizxiiro: i've never heard of "signed pushes" just signed objects (commits, tags, et cetera)15:43
elmikoAJaeger: had a question about the sahara-bandit review, gonna push a new version here shortly15:43
zxiirofungi: I didn't either until 5 minutes ago. I guess it's a shiny new feature.15:43
openstackgerritMichael McCune proposed openstack-infra/project-config: remove sahara-bandit job
elmikoAJaeger: i misunderstood one of your comments, but i think i figured it out15:43
fungizxiiro: oh, wow, i just found the docs for push --signed15:43
*** mtanino has joined #openstack-infra15:43
AJaegerelmiko: ok, will review later15:44
fungizxiiro: so git-review calls `git push` and would presumably need some mechanism to know that it needs to use a different command15:44
elmikoAJaeger: thanks!15:44
zxiirofungi: yeah, in latest Gerrit. It tags your pushes with a "?" if you don't use a signed push.15:44
zxiirofungi: Cool. I think this might be something I can contribute to. I'll poke around at the code.15:44
*** vgridnev has quit IRC15:44
*** IlyaG has joined #openstack-infra15:45
fungizxiiro: also git-review could use some reviewing love. i started going through the oldest open reviews for it a week ago, but didn't get very far15:45
*** eharney has quit IRC15:45
fungialso hashar has asked on the ml if we would tag a new release for it, which i haven't followed up on yet15:46
*** EricGonczer_ has joined #openstack-infra15:46
zxiirofungi: sure think I can help out.15:46
fungiwould be good to see if there are any missing bug fixes of test coverage improvements we should land before making a new release15:46
fungizxiiro: thanks!15:46
fungier, bug fixes OR test coverage improvements15:47
fungifingers gettign ahead of my brain this morning15:47
*** mylu has joined #openstack-infra15:47
*** apoorvad has joined #openstack-infra15:48
*** vgridnev has joined #openstack-infra15:49
*** huzhj_ has quit IRC15:49
*** mylu has quit IRC15:49
*** mylu has joined #openstack-infra15:49
*** kzaitsev_mb has joined #openstack-infra15:49
openstackgerritMichael McCune proposed openstack-infra/project-config: remove sahara-bandit job
*** ihrachys has joined #openstack-infra15:50
*** annegent_ has quit IRC15:51
fungipleia2: StevenK: request on the i18n ml to look into a login problem for a user whose account may have been migrated:
*** jsavak has quit IRC15:53
*** mylu has quit IRC15:53
mnaserhi -infra, not sure if uploads have been still struggling lately, but we have a fix (which is unfortunately pending hardware shipment right now)15:53
*** jsavak has joined #openstack-infra15:53
fungipleia2: StevenK: i'll reply letting them know to make sure they've got an account at but you may have better suggestions15:53
*** keedya has quit IRC15:53
*** annegentle has joined #openstack-infra15:54
*** hashar has joined #openstack-infra15:54
*** keedya has joined #openstack-infra15:56
fungii need to pop out to lunch, but will be back shortly15:56
pc_mHi! Anyone know how I can specify a particular commit for a repo, when using devstack? Have a CI failure, and I want to try to bisect to find commit introducing the failure.15:56
*** sputnik13 has quit IRC15:57
*** ArxCruz has joined #openstack-infra15:58
*** EricGonc_ has joined #openstack-infra15:58
*** ilyashakhat has joined #openstack-infra15:59
*** akshai has quit IRC16:00
*** akshai has joined #openstack-infra16:01
*** eharney has joined #openstack-infra16:01
*** EricGonczer_ has quit IRC16:02
*** mylu has joined #openstack-infra16:02
*** hashar has quit IRC16:02
*** mylu has quit IRC16:03
*** tqtran has joined #openstack-infra16:04
*** maishsk has joined #openstack-infra16:04
*** vgridnev has quit IRC16:04
*** akshai has quit IRC16:05
*** nadya has joined #openstack-infra16:05
jeblairfungi: these are errors talking to git.o.o?  (so ord -> dfw ?)16:05
*** fitoduarte has joined #openstack-infra16:05
*** akshai has joined #openstack-infra16:06
*** jamesmcarthur has quit IRC16:06
*** piet has quit IRC16:07
*** piet has joined #openstack-infra16:07
pleia2fungi: thanks16:07
*** vtapia has joined #openstack-infra16:07
*** mylu has joined #openstack-infra16:08
jeblairfungi: 160328-iad-0001210 submitted16:10
pabelangerand back16:10
pabelangerjust catching up on backscroll16:10
*** ilyashakhat has quit IRC16:12
*** ilyashakhat_mobi has joined #openstack-infra16:12
*** kzaitsev_mb has quit IRC16:12
*** dimtruck is now known as zz_dimtruck16:13
openstackgerritSergey Reshetnyak proposed openstack-infra/project-config: Make gate-sahara-tests-python34 job voting
*** zeih has joined #openstack-infra16:14
*** yamahata has joined #openstack-infra16:15
*** tiswanso has quit IRC16:16
*** e0ne has quit IRC16:16
*** tonytan4ever has joined #openstack-infra16:18
openstackgerritDoug Hellmann proposed openstack-infra/project-config: remove pypi publishing jobs for cloudkitty
*** zeih has quit IRC16:19
openstackgerritBen Nemec proposed openstack-infra/project-config: Add stable branch jobs for tripleo-ci
openstackgerritJames E. Blair proposed openstack-infra/zuul: DNM: Revert "Dependencies from undefined projects"
jeblairfungi, pabelanger: assuming that passes tests, when we restart zuul, let's manually install that commit and run it ^16:20
*** piet has quit IRC16:21
*** jamesmcarthur has joined #openstack-infra16:22
*** esikachev has joined #openstack-infra16:23
nibalizergood morning16:24
nibalizerI have returned from vacation16:24
*** fawadkhaliq has joined #openstack-infra16:25
*** ilyashakhat_mobi has quit IRC16:25
jeblairnibalizer: welcome back!  i hope you enjoyed it!16:27
nibalizeri did16:27
*** fedexo has joined #openstack-infra16:27
*** gyee has joined #openstack-infra16:28
*** tiswanso has joined #openstack-infra16:28
*** yamamoto has joined #openstack-infra16:28
pabelangernibalizer: welcome back16:29
nibalizerthanks !16:29
pabelangerjeblair: WFM16:29
nibalizerhow have things been here?16:29
nibalizerI've been intentionally not paying attention16:29
pabelangerwere off devstack-trusty now16:29
pabelangerand centos7 wheel mirror online16:29
pabelangerthat all I have16:30
*** Sukhdev has joined #openstack-infra16:30
pabelangerand I started this over the weekend:
pabelangersurprisingly ubuntu-xenial dibs are looking promising16:31
*** cody-somerville has quit IRC16:31
*** LaT0rtue_ has quit IRC16:31
*** cody-somerville has joined #openstack-infra16:31
*** cody-somerville has quit IRC16:31
*** cody-somerville has joined #openstack-infra16:31
*** jsavak has quit IRC16:31
*** hieulq has joined #openstack-infra16:31
*** tphummel has quit IRC16:31
*** abregman is now known as abregman|afk16:32
greghayneswhy surprisingly? ;)16:32
*** mylu has quit IRC16:32
*** jsavak has joined #openstack-infra16:32
pabelangerjeblair: which account is rax does infra use for support issues?16:32
pabelangergreghaynes: not as much diskimage-builder, but our puppet manifests16:32
pabelangerminimal changes so far16:32
*** esikachev has quit IRC16:33
*** jsavak has quit IRC16:33
*** armax has quit IRC16:33
*** jsavak has joined #openstack-infra16:33
pabelangergreghaynes: so, found an issue with the base element and dkms last night.  Not sure the best way to fix it.  Basically, dkms on centos7 is epel, which base does not depend on.  I don't think I can add the epel element directly since it is yum syntax16:33
jeblairpabelanger: generally the account tied to whichever servers are having the issue.  however, in this case, i spazzed out and filed the ticket under openstackci instead of openstackjenkins like i should have.16:34
greghaynespabelanger: eek. If that can be moved out of the base element then that would be awesome16:34
pabelangerjeblair: Ah, make sense16:34
jeblairpabelanger: we'll see if that's an issue :)16:34
pabelangergreghaynes: Ya, was going to ping ianw about later today. And get his thoughts16:34
greghaynespabelanger: as in, make centos/rhel/fedora all depend on a dkms element which does all the things it does need to16:34
pabelangerI've worked around it for now16:34
*** zz_dimtruck is now known as dimtruck16:35
pabelangergreghaynes: right16:35
*** mylu has joined #openstack-infra16:35
*** abregman|afk has quit IRC16:36
*** dizquierdo has joined #openstack-infra16:36
*** yamamoto has quit IRC16:36
*** annegentle has quit IRC16:40
*** earlephilhower has quit IRC16:42
*** david-lyle_ has joined #openstack-infra16:44
*** david-lyle has quit IRC16:44
jeblairclarkb, mordred: the nodepool-shade change passes tests and has a +2 from me.
*** claudiub|2 has quit IRC16:45
*** ArxCruz has quit IRC16:46
*** jerryz has joined #openstack-infra16:46
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI
*** maishsk has quit IRC16:47
*** Nikolay_St has quit IRC16:47
openstackgerritAndreas Jaeger proposed openstack-infra/system-config: Remove unused nspr dependency
*** ihrachys has quit IRC16:47
*** Guest21263 has quit IRC16:48
*** david-lyle has joined #openstack-infra16:48
*** armax has joined #openstack-infra16:48
*** jamesmcarthur has quit IRC16:48
openstackgerritAndreas Jaeger proposed openstack/requirements: Remove libnspr4-dev
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with non-HA jobs
*** ddieterly is now known as ddieterly[away]16:48
*** fedexo has quit IRC16:49
*** david-lyle_ has quit IRC16:49
*** LaT0rtue has joined #openstack-infra16:49
mordredjeblair: woot!16:49
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove libnspr4-dev and nspr-devel
*** tphummel has joined #openstack-infra16:50
AJaegerpabelanger, fungi: Just removing libnspr4-dev is a large cleanup ;(
*** jamesmcarthur has joined #openstack-infra16:52
*** annegentle has joined #openstack-infra16:52
*** skolekonov is now known as skolekonov|away16:53
*** ddieterly[away] is now known as ddieterly16:54
pabelangerAJaeger: wow, that is a lot of places16:55
*** sputnik13 has joined #openstack-infra16:55
AJaegerpabelanger: yeah ;(16:55
*** maishsk has joined #openstack-infra16:56
*** Swami has joined #openstack-infra16:57
*** jsavak has quit IRC16:58
jrollfungi: would it be too difficult to be able to hold a node in rax-ord that exhibits network badness?16:58
*** jamesmcarthur has quit IRC16:58
AJaegerjroll: I just run into one like - but I fear it's too late to notice these kind of errors16:59
jrollAJaeger: yeah, that was my thought, too fast between fail and delete16:59
AJaegerThere's no long timeout, it's a direct refusal16:59
AJaegerjroll: exactly16:59
*** fitoduarte has quit IRC16:59
*** asselin has quit IRC16:59
*** asselin has joined #openstack-infra17:00
*** tonytan4ever has quit IRC17:00
*** jamesmcarthur has joined #openstack-infra17:00
*** tonytan4ever has joined #openstack-infra17:01
*** piet has joined #openstack-infra17:01
*** salv-orlando has joined #openstack-infra17:02
*** ddieterly is now known as ddieterly[away]17:02
*** dizquierdo is now known as dizquierdo_afk17:04
*** arxcruz has joined #openstack-infra17:04
*** ociuhandu_ has joined #openstack-infra17:05
*** ociuhandu has quit IRC17:05
*** ociuhandu_ is now known as ociuhandu17:05
*** yamahata has quit IRC17:07
*** vgridnev has joined #openstack-infra17:07
*** yamahata has joined #openstack-infra17:07
openstackgerritJay_Clark proposed openstack-infra/irc-meetings: As discussed with the team, our new timeslot is 1300UTC.  This commit changes our meeting time, and sets the channel to openstack-meeting-4.
*** jamesmcarthur has quit IRC17:11
*** cj has quit IRC17:13
*** EricGonc_ has quit IRC17:14
*** cj has joined #openstack-infra17:15
jeblairfungi: do you know if clarkb is away today?17:15
*** salv-orlando has quit IRC17:16
*** SumitNaiksatam has quit IRC17:16
*** mylu has quit IRC17:16
*** mylu has joined #openstack-infra17:18
fungii am back now, and do not know clarkb's whereabouts17:18
*** mylu has quit IRC17:19
*** jsavak has joined #openstack-infra17:19
*** oanson has joined #openstack-infra17:19
fungijeblair: yes, those were nodes matching network errors reaching for a variety of operations covered by the query at
fungithanks for submitting 160328-iad-000121017:20
*** piet has quit IRC17:21
*** achanda has joined #openstack-infra17:22
fungijeblair: jenkins reports py27 unit test failures on 29830817:22
*** edmondsw has joined #openstack-infra17:23
fungihowever, debug output for zuul unit test failures is so massive it's taking me a bit to retrieve and parse17:23
jeblairfungi: yeah, i'm fixing 308 up now17:23
fungiokay, great. i'll stop investigating ;)17:23
fungiwow, still downloading17:23
jeblairfungi: the connections change had a slight entanglement with foreign projects that i didn't catch in the revert17:24
*** mylu has joined #openstack-infra17:24
*** david-lyle has quit IRC17:24
*** sridhar_ram has joined #openstack-infra17:24
fungiquantum entanglement? (he asks, hopefully)17:25
*** smarcet has quit IRC17:25
fungii think openstack is no longer quantum entangled. we renamed all that17:25
*** esikachev has joined #openstack-infra17:25
*** david-lyle has joined #openstack-infra17:25
AJaegerfungi, could you review my documentation of other-requirements when you have some time, please?
*** Hal has joined #openstack-infra17:26
clarkbjeblair: slow start today17:26
*** Hal is now known as Guest7925517:26
*** achanda has quit IRC17:27
fungiclarkb: i recommend switching to a backoff algorithm if the day does not proceed according to base protocol17:27
clarkbya... not sure how much I will end up being around today17:27
*** cloudtrainme has quit IRC17:27
clarkbit has been very meh so far17:28
*** lakshmiS has joined #openstack-infra17:28
openstackgerritJames E. Blair proposed openstack-infra/zuul: DNM: Revert "Dependencies from undefined projects"
jeblairclarkb: okay; i'm trying to determine the feasibility of landing the nodepool-shade change today17:29
*** IlyaG has quit IRC17:29
jeblairclarkb: i'm here for the next 5.5 hours and could babysit it if we could land it soon.  but i don't know what your review status is on it.17:30
clarkbjeblair: I had managed to review most of the change the other day, I still hadn't gotten through the entire set of provider manager changes17:30
clarkbI can try doing that this morning, but wouldn't count on me to babysit17:30
pabelangerI have some spare cycles to help today, if needed17:32
AJaegerpabelanger, sdague: jd__ confirmed that nspr is not used anymore, could you put this cleanup changes on your review list for the next days, please?
*** Goneri has joined #openstack-infra17:34
*** BigWillie has joined #openstack-infra17:36
*** pvaneck has joined #openstack-infra17:37
jeblairclarkb: cool, i think if you can manage to do that before 2000utc i can probably put it into production then with enough time to back it out if there are problems before i eod.17:37
fungii can commit to reviewing it if more reviews are needed, and expect to be around at least until 2200 or so17:38
fungibut also have other things i can work on if it's in hand17:38
jeblairfungi: thanks; up to you -- i singled out clarkb because he had previous comments on it.17:39
fungiyep, preferred if he has time, but real life sometimes intercedes17:39
*** dtantsur is now known as dtantsur|afk17:39
fungii'll try to give it a peek shortly17:40
clarkbI am catching up on the recent patches now then will review the prvider manager diff17:40
*** esker has quit IRC17:40
openstackgerritAndreas Jaeger proposed openstack-infra/gerrit: Fix .gitreview file
*** jamesmcarthur has joined #openstack-infra17:41
*** hieulq has quit IRC17:41
*** piet has joined #openstack-infra17:43
openstackgerritAndreas Jaeger proposed openstack-infra/gerrit: List system dependencies for running common tests
*** doug-fis_ has quit IRC17:43
lakshmiSAJaeger: experimental worked. Thanks!!! Should i add another patch moving experimental to acutal?17:44
*** krtaylor has quit IRC17:44
*** nelsnels_ has joined #openstack-infra17:46
*** achanda has joined #openstack-infra17:46
*** abe_music has joined #openstack-infra17:46
*** abe_music has quit IRC17:47
*** tongli has joined #openstack-infra17:47
*** kzaitsev_mb has joined #openstack-infra17:47
*** nelsnelson has quit IRC17:48
AJaegerlakshmiS: yes17:49
*** arxcruz has quit IRC17:50
*** akshai has quit IRC17:51
*** akshai has joined #openstack-infra17:52
*** tqtran has quit IRC17:53
*** IlyaG has joined #openstack-infra17:53
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual: Document other-requirements
AJaegerfungi, thanks for review. I reworked a bit ^17:53
fungiAJaeger: awesome, taking another look momentarily17:54
*** krtaylor has joined #openstack-infra17:55
*** piet has quit IRC17:56
*** Sukhdev has quit IRC17:57
AJaegerfungi, I'm experimenting a bit with other-requirements - or cleanup of bindep-fallback. I found one completely unused package - see - and start using it for documentation projects.17:57
*** e0ne has joined #openstack-infra17:57
* AJaeger will continue as time permits17:57
AJaegerBut enough of that for today17:57
fungiAJaeger: very cool find. i have a feeling some things fell out of use and were simply copied from the lists in our puppet manifests because i lacked time to research them all17:58
AJaegerfungi, yep. Similar with asciidoc: Removed from manuals long time ago. Gerrit is only user today...17:59
AJaegerI'll be off for now...17:59
fungihave a good evening AJaeger!17:59
*** jamesmcarthur has quit IRC17:59
AJaegerthanks, fungi18:00
*** baoli has quit IRC18:00
*** tiswanso has quit IRC18:00
AJaegerOne quick question: Where do we install maven? I was surprised to see it on the nodes but couldn't find a reference?18:00
clarkbjeblair: +2'd but not approved, will let you do that when you are ready. The test logs looked good from what I could see too, though I didn't count api calls18:00
*** baoli has joined #openstack-infra18:01
*** jsavak has quit IRC18:01
*** tiswanso has joined #openstack-infra18:01
*** jamesmcarthur has joined #openstack-infra18:02
*** SumitNaiksatam has joined #openstack-infra18:02
*** jsavak has joined #openstack-infra18:02
*** ddieterly[away] has quit IRC18:03
AJaegeroff now, will read answer in backscroll later ;)18:03
*** Jeffrey4l has quit IRC18:04
*** annegentle has quit IRC18:04
*** doug-fish has joined #openstack-infra18:04
*** piet has joined #openstack-infra18:05
*** doug-fis_ has joined #openstack-infra18:05
*** vgridnev has quit IRC18:06
jeblairclarkb: thanks, aprvd, will restart after lunch18:07
pabelangerGoing to give it a test run locally too18:08
jeblairinfra-root: i will be periodically afk for a bit this week: traveling to/attending board meeting tuesday; conferencing wednesday; traveling home thursday.18:08
*** oanson has quit IRC18:08
jeblair(it's in my backyard so i figured i'd pop in)18:09
*** doug-fish has quit IRC18:09
openstackgerritgreghaynes proposed openstack/diskimage-builder: dhcp-all-interfaces depends on dhcp
openstackgerritLakshmi N Sampath proposed openstack-infra/project-config: Change experimental to actual for searchlight
openstackgerritMerged openstack-infra/nodepool: Use shade for all OpenStack interactions
pleia2jeblair: have fun18:11
*** Rockyg has joined #openstack-infra18:12
*** smarcet has joined #openstack-infra18:12
*** david-lyle_ has joined #openstack-infra18:12
fungiyou're hosting a conference in your back yard? awesome!18:12
*** david-lyle has quit IRC18:14
jeblairwell, someone else is hosting it, which is kind of weird i guess18:15
*** zeih has joined #openstack-infra18:15
*** degorenko is now known as _degorenko|afk18:15
*** pcaruana has quit IRC18:16
*** sbalukoff has joined #openstack-infra18:16
fungidon't forget to keep plenty of pickle relish on hand18:16
*** salv-orlando has joined #openstack-infra18:18
*** amuller has joined #openstack-infra18:18
amullerGood day everyone. Time for my weekly 'Is it just me or everyone else?'. We're seeing lots of git timeouts in various Neutron jobs.18:18
amullerAnd so I wanted to ask if ya'all are seeing it anywhere else, if it's a known issue, etc18:19
*** ddieterly has joined #openstack-infra18:19
*** zeih has quit IRC18:19
jeblairamuller: yeah, seems to be problems in one particular region, so we're in the process of shutting it off:
*** ArxCruz has joined #openstack-infra18:20
*** ddieterly is now known as ddieterly[away]18:20
amullerjeblair: You're a saint, thank you18:20
amullerkevinbenton: ^18:20
openstackgerritMerged openstack-infra/shade: Add release notes for new create_image_snapshot() args
fungithey're running openstack, so you know... all the usual sausage factory analogies applie18:21
fungiwhat an odd typo18:22
pabelangerhopefully this is duped, lots connection to znc18:22
pabelangeramuller: Yup, ticket with rax already created18:22
pabelangeramuller: you can also check too in the future18:22
*** ddieterly[away] is now known as ddieterly18:22
amullerchecking would be a good idea in general in these cases I suppose18:23
*** jamesmcarthur has quit IRC18:23
mordredjeblair: I'm around, although on a plane - so I can help look at any issues that arise if they do18:23
amullersometimes I'm one of the first people to notice new failures so I often pop by in here to ask18:23
mordredjeblair: and I'm sure Shrews is excited to help too :)18:23
*** IlyaG has quit IRC18:23
*** edmondsw has quit IRC18:23
mordredShrews: (btw, shade nodepool patch lnaded)18:23
jeblairmordred, Shrews: cool.  i'm planning on restarting nodepool with it around 20:00utc.  i'm prepared to babysit it myself, but figure if either/both of you are around that's ++.18:24
Shrewsmordred: jeblair: so excited18:24
*** ArxCruz has quit IRC18:24
fungiearly morning on summit wednesday! we have an infra fishbowl booked for 9am!18:24
*** salv-orlando has quit IRC18:25
jeblairfungi: i'd like to report your non-standard use of "!".18:25
Shrewsjeblair: that's dinner time for me (if i did the conversion correctly), but will attempt to have my laptop close by18:25
fungijeblair: stuck key, what can i say!18:25
openstackfungi: Error: "!!1!eleven" is not a valid command.18:25
jeblairShrews: it's ~1.5 hours from now18:25
fungii know openstackbot, i know...18:26
Shrewsjeblair: oh. i failed at conversion18:26
jeblairShrews: it's more like your tea-time, which i know is important to you.  :)18:26
mordredjeblair: I expect one of four outcomes, btw:18:26
openstackgerritEric Brown proposed openstack-infra/project-config: Remove oslo.messaging from bandit integration
openstackgerritLakshmi N Sampath proposed openstack-infra/project-config: Change experimental to actual for searchlight
*** EricGonczer_ has joined #openstack-infra18:26
Shrewsjeblair: it IS important. but i will be here then18:27
fungimordred: we're using a two-bit analysis now?18:27
jeblairmordred: fail; fail; success; success?18:27
jeblairfungi, mordred: big fail; little fail; little success; big success?18:27
jeblair"ba da boom"  "big ba da boom"18:28
mordredjeblair: 1) it works perfectly fine and we notice essentially nothing 2) it works fine, but is making more API calls than we'd like, but we can fix by doing analysis and follow up patches 3) it breaks, but the breaks and easy/obvious and we can roll proper fixes in quickly 4) it breaks stupendously and we have to revert18:28
mordredso, yes18:28
openstackgerritMerged openstack-infra/project-config: Turn off rax-ord
jeblairmordred: clarkb did mention he didn't count api calls; so we could be surprised there :)18:28
fungior just switch to hard liquor18:29
clarkbpart of that is firefox has really bad search counting now18:29
mordredjeblair: yah. I mean, I believe we've rectified the api call count imbalance from last time18:30
clarkb"more than 100 matches"18:30
clarkbwhy not give me a number...18:30
mordredbut also, I'm hoping that if we missed something it's not so bad that we need to react harshly and can just work it18:30
clarkb might get annoying but we dont shade at debug level in prod18:30
*** pcaruana has joined #openstack-infra18:30
jeblairclarkb: it's the same misguided thinking that gets us "about 10 minutes ago"18:30
*** EricGonczer_ has quit IRC18:31
mordred(1 extra call per server - annoying, but not crippling - 100 extra calls per server, obviously unworkable)18:31
*** jaypipes has quit IRC18:31
fungii think gerrit is transitioning to lapine counting (one, two, three, hrair, hrair, hrair, hrair...)18:31
*** EricGonczer_ has joined #openstack-infra18:32
*** vgridnev has joined #openstack-infra18:33
*** baoli has quit IRC18:33
*** kdas_ has joined #openstack-infra18:33
*** tqtran has joined #openstack-infra18:34
*** sdake_ has joined #openstack-infra18:34
*** sdake_ has quit IRC18:35
*** kushal has quit IRC18:35
clarkbthe thing to grep for is "running task" and compare that to teh run fro shadification and the runs prior18:35
*** baoli has joined #openstack-infra18:35
clarkbkeep in mind the shadification change adds in some extra operations so it won't be a direct coparison18:35
*** keedya has quit IRC18:36
*** sdake has quit IRC18:37
*** dizquierdo_afk is now known as dizquierdo18:37
zarofungi: sorry i wasn't around for the bad gerrit repo.  did you ever get the log dump to see if we can narrow down that problem?18:39
*** sdake has joined #openstack-infra18:39
*** baoli has quit IRC18:40
fungizaro: i have the stack trace, meant to put it into a paste... just a moment18:41
*** knikolla_ has joined #openstack-infra18:41
*** baoli has joined #openstack-infra18:41
*** fitoduarte has joined #openstack-infra18:43
*** doug-fis_ has quit IRC18:43
zarojeblair: the change that wasn't searchable, is actually due to a bug with cherry-pick in Gerrit 2.4,
*** cloudtrainme has joined #openstack-infra18:44
*** doug-fish has joined #openstack-infra18:44
openstackgerritJames Slagle proposed openstack-infra/project-config: Add a periodic mitaka job for TripleO
*** sdake has quit IRC18:44
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add support for mitaka jobs
pabelangerinstalling latest nodepool locally now18:44
zarojeblair: might be able to fix that that by manually creating the ref but not sure if it's worth doing.18:45
openstackgerritJames Slagle proposed openstack-infra/project-config: Add a periodic mitaka job for TripleO
*** baoli has quit IRC18:46
*** sdake has joined #openstack-infra18:48
*** baoli has joined #openstack-infra18:48
*** e0ne has quit IRC18:50
*** achanda has quit IRC18:51
*** jamesmcarthur has joined #openstack-infra18:51
*** esikachev has quit IRC18:52
*** achanda has joined #openstack-infra18:54
dimsAJaeger : can you please cross check my answer in  (string translations!)18:54
*** fawadkhaliq has quit IRC18:55
*** fawadkhaliq has joined #openstack-infra18:55
*** fitoduarte has quit IRC18:56
*** baoli has quit IRC18:58
*** baoli has joined #openstack-infra18:59
*** baoli has quit IRC19:00
*** brad_behle has joined #openstack-infra19:01
*** tiswanso has quit IRC19:01
*** edmondsw has joined #openstack-infra19:01
*** sridhar_ram1 has joined #openstack-infra19:01
*** sridhar_ram has quit IRC19:02
*** piet has quit IRC19:02
*** tiswanso has joined #openstack-infra19:02
*** jamesmcarthur has quit IRC19:08
pabelangerjeblair: mordred: rebuild of local nodepool looks good. Everything came online as expect in my local cloud19:08
*** earlephilhower has joined #openstack-infra19:10
openstackgerritCedric Brandily proposed openstack-infra/shade: Support InsecureRequestWarning == None
mordredpabelanger: WOOT!19:10
*** EricGonczer_ has quit IRC19:10
*** jamesmcarthur has joined #openstack-infra19:10
pabelangerYup, another data point!19:10
*** annegentle has joined #openstack-infra19:13
*** IlyaG has joined #openstack-infra19:14
*** tphummel has quit IRC19:17
*** ZZelle has quit IRC19:18
*** e0ne has joined #openstack-infra19:19
*** nadya has quit IRC19:19
*** e0ne has quit IRC19:21
*** angdraug has joined #openstack-infra19:22
*** doug-fis_ has joined #openstack-infra19:22
*** brad_behle has left #openstack-infra19:23
*** IlyaG has quit IRC19:24
openstackgerritMerged openstack/requirements: Remove libnspr4-dev
*** doug-fish has quit IRC19:25
*** roxanaghe has quit IRC19:25
*** e0ne has joined #openstack-infra19:26
*** Sukhdev has joined #openstack-infra19:27
*** annegentle has quit IRC19:27
*** doug-fis_ has quit IRC19:27
*** doug-fish has joined #openstack-infra19:28
*** e0ne has quit IRC19:28
*** brad_behle has joined #openstack-infra19:31
*** jamesmcarthur has quit IRC19:31
*** brad_behle has left #openstack-infra19:32
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: improve to edit the constraints file
*** annegentle has joined #openstack-infra19:33
*** roxanaghe has joined #openstack-infra19:35
*** esker has joined #openstack-infra19:35
*** sdake_ has joined #openstack-infra19:36
*** mylu has quit IRC19:36
jeblairzaro: wow, thanks19:36
*** MarkAtwood has joined #openstack-infra19:37
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: improve to edit the constraints file
*** ddieterly is now known as ddieterly[away]19:38
*** mylu has joined #openstack-infra19:38
*** fawadkhaliq has quit IRC19:38
*** sdake has quit IRC19:38
*** fawadkhaliq has joined #openstack-infra19:38
*** ddieterly[away] is now known as ddieterly19:42
*** salv-orlando has joined #openstack-infra19:42
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Do not install php packages
*** weshay_ has joined #openstack-infra19:46
*** Sukhdev has quit IRC19:47
*** baoli has joined #openstack-infra19:47
*** baoli has quit IRC19:48
*** baoli has joined #openstack-infra19:49
*** IlyaG has joined #openstack-infra19:49
*** yamahata has quit IRC19:50
*** weshay has quit IRC19:50
*** BigWillie has quit IRC19:50
*** ZZelle has joined #openstack-infra19:51
*** salv-orlando has quit IRC19:52
*** Rockyg has quit IRC19:53
*** tlbr has quit IRC19:53
*** ajmiller has quit IRC19:54
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Revert "Add script to facilitate project renaming in project-config repo"
*** nadya has joined #openstack-infra19:55
pleia2fungi: thanks for the heads up with the login issue on the i18n list, though I haven't figured out what's going on yet19:55
pleia2the user migration from translate-dev was nothing special19:56
*** ajmiller has joined #openstack-infra19:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Revert "Add script to facilitate project renaming in project-config repo"
*** ArxCruz has joined #openstack-infra19:56
pleia2wondering if it may just be worth deleting their "old" account that we migrated and just having them set something up again, since they haven't used it and there isn't history with that account in production19:57
fungithat seems reasonable19:57
*** knikolla_ has quit IRC19:57
pleia2I'll go ahead and do that, if we run into this again with someone else I'll dive in deeper19:59
*** EricGonczer_ has joined #openstack-infra19:59
*** esker has quit IRC19:59
*** esikachev has joined #openstack-infra19:59
pleia2"This user cannot be removed from the system. You can deactivate the user instead"20:00
*** ArxCruz has quit IRC20:00
Shrewsjeblair: zero hour?20:00
* mordred stares intently at jeblair20:02
* mordred hands Shrews a beer20:02
* mordred screams with excitement20:03
*** sdake has joined #openstack-infra20:03
*** sputnik13 has quit IRC20:03
* mordred hands Shrews a box of baby goats and one mostly not unhappy baby rhino20:03
*** ddieterly is now known as ddieterly[away]20:03
jeblairmordred: please don't get escorted off the plane.  at 35k.20:03
mordredjeblair: it's literally the only thing I strive for on a consistent basis20:03
jeblairnodepool==0.2.1.dev72  # git sha e1f4a1220:04
jeblairthat looks good.20:04
*** Sukhdev has joined #openstack-infra20:04
mordredjeblair: should we tag nodepool the pre-shade commit?20:05
mordredjeblair: I think someone in packaging world asked us to do that at some point20:05
mordredjeblair: (we can do it after we deploy of course)20:05
*** esikachev has quit IRC20:05
mordredor, wait - that was the commit from before we started depending on shade20:05
*** sdake_ has quit IRC20:06
jeblairmordred: well, if we wanted to tag another, no objections here :)20:06
*** tphummel has joined #openstack-infra20:06
mordredmeh. let's just roll20:06
*** maishsk has quit IRC20:06
jeblairmany images are building; this will probably be disruptive for them.20:06
mordredjeblair: you don't need to restart the builders20:06
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: remove pypi publishing jobs for cloudkitty
openstackgerritMark T. Voelker proposed openstack-infra/irc-meetings: Update DefCore Meeting description
jeblairmordred: yeah, but their gearman jobs will be canceled20:07
mordredah. good point20:07
mordredyah. that will likely be disruptive20:07
jeblairit should be handled now though, so ahead we go20:07
jeblairrestarting now20:08
*** nadya_ has joined #openstack-infra20:08
jeblair2016-03-28 20:08:41,615 DEBUG nodepool.ProviderManager: Manager internap-nyj01 ran task FlavorGetExtraSpecs in 0.202835083008s20:08
jeblair2016-03-28 20:08:41,616 DEBUG nodepool.ProviderManager: Manager internap-nyj01 running task FlavorGetExtraSpecs (queue: 6)20:08
jeblairmany of these tasks20:08
pabelangerI can confirm that locally20:09
*** maishsk has joined #openstack-infra20:09
*** kgiusti has left #openstack-infra20:09
*** kzaitsev_mb has quit IRC20:09
*** baoli has quit IRC20:10
*** nadya has quit IRC20:10
*** baoli has joined #openstack-infra20:10
*** akscram has quit IRC20:11
*** baoli has quit IRC20:11
* Shrews sips tea... awaits more info20:11
jeblairi wonder if that just happens on startup?20:11
*** baoli has joined #openstack-infra20:11
*** akscram has joined #openstack-infra20:11
jeblairor if that's each time we create a server20:12
jeblair2016-03-28 20:13:01,956 DEBUG nodepool.ProviderManager: Manager osic-cloud1 ran task ServerList in 0.681684970856s20:13
jeblair2016-03-28 20:13:02,801 DEBUG nodepool.ProviderManager: Manager osic-cloud1 ran task ServerList in 0.757560014725s20:13
jeblairthose ran very close together -- is caching working correctly there?20:14
*** yamahata has joined #openstack-infra20:14
*** lakshmiS has quit IRC20:14
*** zeih has joined #openstack-infra20:15
mordredjeblair: it's just at startup20:16
mordredoh - wait. I responded poorly20:16
*** matrohon has joined #openstack-infra20:16
mordredflavor getextraspecs will only run at startup, they're followup calls to the flavor list20:16
mordredthe server list seems close together ... we might be missing a config option ... one sec20:17
jeblairok, so that's just something we should have the provider manager de-dup.20:17
*** BigWillie has joined #openstack-infra20:17
jeblairbut is not high-priority since it's only at startup (or reconfigure, probably)20:17
*** dprince has quit IRC20:17
*** jamesmcarthur has joined #openstack-infra20:18
*** lakshmiS has joined #openstack-infra20:18
Shrewsoh that's neat20:19
*** piet has joined #openstack-infra20:20
*** zeih has quit IRC20:21
*** Sukhdev has quit IRC20:21
*** baoli has quit IRC20:22
jeblairso active questions: 1) caching in serverlist; 2) caching in portlist; 3) the more-than-one-port exception20:22
jeblair(re 2: i'm seeing a lot of portlist tasks as well)20:23
*** automagically has left #openstack-infra20:23
jeblairi suspect the tripleo cloud is out of commission due to #320:23
*** ZZelle has quit IRC20:23
*** lakshmiS has quit IRC20:23
*** ddieterly[away] is now known as ddieterly20:24
*** fawadkhaliq has quit IRC20:24
*** vgridnev has quit IRC20:25
Shrewsi'm trying to figure out where that error message from #3 is coming from20:25
*** fawadkhaliq has joined #openstack-infra20:25
* morgan waves at Shrews.20:25
Shrewsah, there it is20:25
morganmostly cause Shrews is there.20:25
morganjeblair: i might have a couple questions for you and/or mordred re nodepool and caching soon.20:26
Shrewsmorgan: howdy howdy20:26
jeblairShrews:  now with more trackbacks :)20:26
*** esikachev has joined #openstack-infra20:27
*** ybathia has joined #openstack-infra20:27
jeblairi have received an out-of-band message from mordred saying clouds.yaml is missing cache.expiration.server: 520:27
*** Sukhdev has joined #openstack-infra20:28
*** maishsk has quit IRC20:28
jeblairi'll look into that while Shrews looks into the tripleo port thing20:28
*** vgridnev has joined #openstack-infra20:29
*** vgridnev has quit IRC20:29
*** weshay_ has quit IRC20:32
clarkbtripleo hosts have two networks each20:33
clarkbwhich means two ports20:33
*** nadya_ has quit IRC20:33
anteayaI'm still enjoying Easter Monday and good thing too as I probably wouldn't otherwise have witnessed the beginning of the spring flood in my basement20:33
anteayaI had hoped I had missed it this year, but nope20:34
anteayabased on the speed of the water coming in I expect I'll be up every hour all night long20:34
anteayamy attendance tomorrow is questionable20:34
dimsanteaya ouch!20:34
anteayayeah, I just have to stay ahead of the water and replace and dry towels all night20:35
pc_mFor neutron, is there a way to tell what commit on master, corresponds to the commit on stable/mitaka?20:35
*** Nikolay_St has joined #openstack-infra20:37
amullerpc_m: gerrit search works with commit hashes20:37
amullerpc_m: then you click on the patch, then click on its change ID20:37
amullerpc_m: it will bring up the patch including any cherry picks20:37
pc_mamuller: not working with a specific commit/review20:38
*** maishsk has joined #openstack-infra20:38
Shrewsclarkb: ahhh. so we need to pass which network to use for the fip. how would we determine that?20:38
pc_mamuller: We have a CI system that is failing with the latest on master. I think (confirming) it works with stable/mitaka neutron.20:39
fungipc_m: that behavior depends on the backport keeping the change-id commit message header from the master commit20:39
fungithe searching on the change-id depends on it, i mean20:39
pc_mamuller: Want to select the commit on master that corresponds to stable/mitaka and do a git bisect to find the commit that introduced the failure.20:39
clarkbShrews: one of the networks should be annotated public? I forget20:40
pleia2anteaya: yikes, sorry to hear20:40
*** jsavak has quit IRC20:40
fungipc_m: i see, you want to start from the point in master where syable/mitaka branched?20:40
*** asettle has joined #openstack-infra20:40
*** Goneri has quit IRC20:40
amullerpc_m: why don't you go to stable/mitaka branch, look at HEAD, write down the commit has, go back to master, then git bisect with that patch as the 'low' and master HEAD as the 'high'20:40
amullercommit hash*20:41
pc_mfungi: yes20:41
fungiamuller: stable/mitaka may have diverged after branching20:41
amullerfungi: true20:41
pc_mfungi: hmmm20:41
pc_mis there a way to find the point that stable/mitaka branched?20:42
*** Sukhdev has quit IRC20:42
pc_mon master20:42
fungiyou're looking for the last common ancestor, i think20:42
*** sdake_ has joined #openstack-infra20:43
fungii think git merge-base is the tool you probably want20:43
*** sdake has quit IRC20:43
*** matrohon has quit IRC20:43
pc_mfungi: Thanks will look at that.20:43
fungithough the idea of a "last common ancestor" is somewhat fuzzy since git is a dag20:44
pc_mfungi: So looks like git merge-base stable/mitaka...HEAD20:45
fungi`git merge-base origin/stable/mitaka origin/master` says dd4f1253c951d78a5b497680dfb31317ba469a58 which was the merge for "De-dup conntrack deletions before running them"20:46
fungi12 days ago20:46
amrithfungi, ping ...20:46
fungiamrith: hey there20:46
amrithI got to the bottom of one py27 failure20:47
fungidetails on the ironic unit test issues?20:47
amrithRoutes released 2.3 today20:47
amrithand that broke some of trove's gate jobs20:47
amrithcouple of issues there20:47
amrithgetting a list20:47
*** jamesmcarthur has quit IRC20:47
amriththis is the trove issue20:47
openstackLaunchpad bug 1563028 in Trove "Routes version 2.3 broke the way we register routes" [Critical,In progress] - Assigned to Amrith (amrith)20:47
amrithit was exposed because of this (also trove issue)
openstackLaunchpad bug 1563038 in Trove "fix tox.ini in Trove to respect upper-constraints.txt" [High,New] - Assigned to Amrith (amrith)20:48
fungioh lovely20:48
amriththis means that if someone bumps upper-constraints.txt and allows Routes 2.3, then some other projects may also trip over this20:48
amrithI sent this to the ML20:49
amrithas a heads-up20:49
pc_mfungi looks like one uses triple dot.20:49
*** vgridnev has joined #openstack-infra20:49
amrithSo, if you see a failure that looks like:20:50
amrithFile "/home/jenkins/workspace/gate-trove-python27-db/.tox/py27/local/lib/python2.7/site-packages/routes/", line 168, in connect20:50
amrith2016-03-28 18:58:54.390 |     routename, path = args20:50
amrith2016-03-28 18:58:54.390 | ValueError: need more than 1 value to unpack20:50
amriththen you want to check the bugs above :(20:50
pc_mfungi: git log stable/mitaka...HEAD20:50
*** kzaitsev_mb has joined #openstack-infra20:50
fungiamrith: thanks--will be on the lookout. we definitely don't seem to be capping routes in global-requirements.txt20:50
amrithfungi, it is capped in upper-constraints.txt20:50
jeblairclarkb, Shrews: we don't have the 'public' annotation set on either of the tripleo networks; however, i don't think that would solve this problem20:51
amrithbut trove wasn't using that in tox.ini like (for example) neutron is20:51
fungiamrith: right, that's less of a cap and more of a "this is what we're testing with today" list20:51
amrithwhich is why we were the canary in the coal mine.20:51
jeblairclarkb, Shrews, mordred: i'm going to update with our current issues20:51
*** mylu has quit IRC20:51
fungipc_m: intersting, the manpage i have for git-merge-base uses a space separator between ref names20:51
fungipc_m: also i wouldn't trust stable/mitaka and HEAD to be in sync with origin/stable/mitaka and origin/master, better to be explicit20:52
pc_mfungi: yeha, I see someone saying you can do git log A...B20:52
pc_mfungi: Good IDea20:52
fungiyou can, that gives you the list of commits from a to b20:52
*** achanda has quit IRC20:52
fungithough supposed to be .. not ... afaik20:52
*** BigWillie has quit IRC20:52
*** baoli has joined #openstack-infra20:52
amrithanyway fungi ... just wanted to give you a quick update on this. will follow up on the other issue from the email thread. toodle doo for now ... have to go head into traffic.20:52
fungimordred: your plane landed in spain/.20:53
* mordred has found chair20:53
mordredfungi: salt lake city20:53
mordredfungi: same thing really20:53
fungimordred: funny how salt-lakers sound like they're speaking spanish20:53
fungiamrith: thanks again20:53
pc_mfungi thanks!20:54
*** esikachev has quit IRC20:54
*** jsavak has joined #openstack-infra20:54
jeblairmordred, Shrews: so if #3 is going to take some time, i reckon we ought to roll back.  i can just check out the old commit on nodepool.o.o and reinstall and restart manually.20:55
*** eil397 has joined #openstack-infra20:55
Shrewsjeblair: the thing is, if there is more than one network, we need to be telling it which network to use. i do not know how to determine that info20:55
*** kzaitsev_mb has quit IRC20:55
mordredShrews: yah20:56
mordredso - my thought is that we use one of the "I have told you what my network is" settings20:56
jeblaircould we re-use nodepool's "public" network attribute for this?  it seems like the case to me that if we are attaching floating ips in nodepool, that's always going to be on the "public" network....20:57
*** MarkAtwood has quit IRC20:57
*** efoley has joined #openstack-infra20:57
mordredthat would actually break a different logic20:57
Shrewsmordred: there might be a shade bug, too, where this is still going to happen b/c we do not pass the fixed address through to the _get_free_fixed_port() call20:58
mordredwhich is that for internap we specify the 'public' network since there are two and we don't know the difference20:58
mordredand shade says "do I have an ip on the network you've told me is public? if so, we're good. if not, we need a floating ip"20:58
mordredjeblair: so I actually think we should use the internal_network setting20:59
mordredwhich is how you tell shade which network is you "private" network20:59
mordredwe could also add a whole new thing of "which network should your floating ips NAT to"20:59
*** fawadkhaliq has quit IRC20:59
* mordred has no strong feelings and is open to discussing shed paint20:59
efoleyHi, are there some issues with jenkins? I'm getting random failures on different tests are failing on each jenkins run.21:00
Shrewsjeblair: my recommendation is rollback21:00
*** fawadkhaliq has joined #openstack-infra21:00
*** amuller has left #openstack-infra21:00
fungisounds like this is probably a shed which should be painted in comfort after rolling back, then trying again later, agreed21:00
*** baoli has quit IRC21:00
mordredyah. I think each of these are very understandable and tractable problems21:00
fungirushing a new feature into shade so we can get tripleo back online seems like not a great set of constraints to impose on the dev process21:01
mordredbah. it's how all good software is writen - for some values of good, software, all and written21:01
jeblairmordred: how would using public break internap?  internap will have the ip already so we still won't need to add a floating ip.21:01
mordredit'll break tripleo21:01
jeblair(i'm performing the rollback now)21:01
mordredjust differently21:01
mordredbecause the do-i-need-a-floating-ip logic will stop at "yes, you have an ip from the network you've defined as public"21:02
*** ericksonsantos has joined #openstack-infra21:02
*** jamesmcarthur has joined #openstack-infra21:03
*** |-paul-| has joined #openstack-infra21:03
jeblairmordred: we have an ip on that network before creating a floating ip?21:03
*** thorst has quit IRC21:03
*** IlyaG has quit IRC21:03
*** achanda has joined #openstack-infra21:04
jeblair[oldpool restarted]21:04
*** Sukhdev has joined #openstack-infra21:04
*** IlyaG has joined #openstack-infra21:04
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add server and port caching settings for nodepool
jeblairoh i was writing that21:05
mordredjeblair: yes. configuring a public network is the way we tell shade that even though it cannot infer which of the two networks is public and directly routable, this is the one21:05
*** rockyg has joined #openstack-infra21:05
mordredjeblair: oh, sorry21:05
* jeblair updates etherpad to assign that task to mordred21:05
*** Sukhdev has quit IRC21:05
jeblairmordred: what ip address do we have assigned to a tripleo node on the ostensibly public network before we get a floating ip?21:06
*** baoli has joined #openstack-infra21:07
mordredjeblair: let me start over - I can't come up with a way to answer that question that is not more confusing21:07
*** Sukhdev has joined #openstack-infra21:08
*** thorst has joined #openstack-infra21:08
mordredjeblair: configuring which network is public has a current meaning - that meaning is that a network labeled in config as public is a network from which one expects to get directly attached IP addresses21:08
*** kzaitsev_mb has joined #openstack-infra21:08
mordredjeblair: so if we configure a network as public, and shade finds that the server has an ip from that network, it will consider its job done, becaue the server has an ip from the public network21:09
jeblairmordred: that is not its meaning in nodepool21:09
jeblairmordred: in nodepool that means "this network is attached to the internet; therefore, the nodepool server should be able to talk to ip addresses on that network (howevery they are obtained)"21:10
mordredgotcha. so we are definitely talking about different things. that's good - it will make this better21:10
mordredI'm talking about the external_network setting in clouds.yaml, which has the above semantics21:11
jeblairmordred: okay.  i did say "re-use nodepool's "public" network attribute for this" :)21:11
jeblairmordred: i still don't understand even so why we would have an ip on a network that requires floating ips before we have created a floating ip21:12
clarkbthsts a neutronism21:12
clarkbbecause nat21:12
*** thorst has quit IRC21:12
_odyOk.  Can't tell if anything strange is happening so I am just going to mention it...cloning anything from is taking me 15 ore more and occasionally just fails.21:13
*** aysyd has quit IRC21:13
bkero_ody: are you comcasting?21:13
_odybkero: No.21:13
bkeroAh, I'm also having connectivity issues to multiple places today21:14
bkero(on comcast)21:14
_odyGoing out HE IPv4 right now.21:14
pabelanger_ody: bkero: Know issue today, see
*** EricGonczer_ has quit IRC21:14
*** EricGonczer_ has joined #openstack-infra21:15
*** tqtran has quit IRC21:16
*** ldnunes has quit IRC21:16
*** fitoduarte has joined #openstack-infra21:16
bkerowell then21:17
bkeropabelanger: thanks21:17
*** eharney has quit IRC21:17
*** tlbr has joined #openstack-infra21:18
_odybkero: Does that link also just hang or come back with no data for you?21:18
*** julim has quit IRC21:18
fungi_ody: pabelanger: as far as i know that should have been limited to rackspace's ord region21:19
*** piet has quit IRC21:19
bkero_ody: it sits there with no apparent update for a while, then shows graphs21:19
*** aeng has joined #openstack-infra21:19
bkeropresumably the gifs are generated on pageload and they're not too quick21:19
fungimost recent hit on that query seems to be several hours ago21:20
fungiif you add the build_node column you'll see the last hit for those errors was on a worker which ran in rax-ord, before we turned out use of that region down to 021:21
*** ociuhandu has quit IRC21:21
fungier, turned our use of that region down to 021:21
*** prometheanfire has quit IRC21:21
*** prometheanfire has joined #openstack-infra21:22
dougwigmy nodepool suddenly decided it needed to start assigning floating ips on a network that doesn't support them. known issue?21:23
*** _ari_ has quit IRC21:23
fungidougwig: did you restart after merged?21:24
fungidougwig: if so, you may want to roll that back. a revert is on the way21:24
dougwigfungi: yes, server just rebooted.21:24
dougwigfungi: ahh, ok, thanks.21:24
* dougwig goes to disable puppet cron and rollback.21:25
*** thorst has joined #openstack-infra21:26
Shrewsdougwig: that's interesting. mordred, perhaps we should have a config option to turn off/on auto_ip? ^^^21:27
*** EricGonczer_ has quit IRC21:27
*** sdake_ is now known as sdake21:27
*** bpokorny has quit IRC21:27
efoleyHi, are there some issues with jenkins? I'm getting random failures on different tests are failing on each jenkins run.21:27
*** salv-orlando has joined #openstack-infra21:27
dougwigShrews: used to be that marking the network 'public' punted that; did that change?21:28
*** sridhar_ram1 is now known as sridhar_ram21:29
*** thorst has quit IRC21:30
*** bpokorny has joined #openstack-infra21:30
fungiefoley: helps to characterize how they're failing and look for commonality21:31
*** jamesmcarthur has quit IRC21:31
Shrewsdougwig: well, the meaning of that word was just discussed in scrollback. i think we need some shade changes before trying this again21:31
*** jamesmcarthur has joined #openstack-infra21:31
efoleyfungi: It's the same testcase that fails, but different tests each time21:32
dougwigShrews: ok.  fwiw, my use case is a provider net that's directly accessible, which do not support floating ips (which I personally think is good, because using floating IPs in the common case is maddening.)21:32
fungiefoley: strangely, the change is modifying a devstack plugin, yet its the unit tests failing21:33
*** jsavak has quit IRC21:33
*** ybathia has quit IRC21:33
efoleyfungi: Worked perfectly before, and now... random failures! That's what's weird about it21:33
_odyfungi: All my clones from are still taking forever and/or failing so there might me something else going on.  I've switched to sourcing github and everthing is fine to do local testing again.21:33
dougwigfungi: many thanks, humming along now.21:34
pabelanger_ody: fungi: which protocol are you using? http(s) / git ?21:34
fungi_ody: could it be ipv6 connectivity that's your problem, or are you on an ipv4-only connection?21:34
*** piet has joined #openstack-infra21:34
*** jamesmcarthur has quit IRC21:34
fungi_ody: any silly corporate proxies to contend with?21:35
*** salv-orlando has quit IRC21:35
*** ociuhandu has joined #openstack-infra21:35
*** regXboi has quit IRC21:35
*** doug-fish has quit IRC21:36
*** Sukhdev has quit IRC21:36
*** achanda has quit IRC21:37
*** Sukhdev has joined #openstack-infra21:37
*** salv-orlando has joined #openstack-infra21:38
*** tqtran has joined #openstack-infra21:39
_odypabelanger: https21:40
_odyfungi: Was on ipv4 only, inside and outside an openstack instance.  No proxies.21:40
zarofungi: git-review wip on push feature is just a generic way to support Gerrit's feature to set labels on push.  that's a feature in gerrit21:41
*** salv-orlando has quit IRC21:42
*** esikachev has joined #openstack-infra21:42
*** tiswanso has quit IRC21:43
*** zeih has joined #openstack-infra21:43
*** ddieterly is now known as ddieterly[away]21:44
pabelanger_ody: So, as a data point, we've seen poor https performance until https with centos6: bug 1237395 in git "extremley slow https clone performance" [Unspecified,Assigned] - Assigned to pstodulk21:44
fungizaro: yep, i haven't looked at it in a while. it doesn't actually say "work in progress" or hard-code any specific labels/values anywhere now, i guess?21:44
*** salv-orlando has joined #openstack-infra21:45
*** cfarquhar has quit IRC21:45
fungizaro: better not to call it a "wip on push" feature in that case, just "vote on push" or something21:45
zarofungi: yeah, it's just setting vote on existing labels in gerrit.21:45
zarocommit msg is probably a little misleading21:46
fungiwe happen to use a custom label+value to indicate work in progress, but i don't want that encoded in git-review anywhere21:46
zarofungi: +1 there, and it's not in that change.21:47
*** cfarquhar has joined #openstack-infra21:48
*** cfarquhar has quit IRC21:48
*** cfarquhar has joined #openstack-infra21:48
zaroyep, even the example uses Code-Review label.  only mention of wip is in the commit message21:48
fungithat said, let's not land more feature-like changes until after we tag a release, and just flush lingering bug fixes and test improvements first21:48
*** zeih has quit IRC21:49
jeblairyeah, i think it might pass the sniff test since it's using a generic feature in gerrit, and it's not available some other way (if you want to set a label on push, you would need git-review to support this).21:49
fungiefoley: i was going to try to recreate that failure on my system, but seems to need libvirt to be able to run unit tests21:49
jeblairbut also, not urgent before tagging :)21:49
*** |-paul-| has quit IRC21:50
*** ybathia has joined #openstack-infra21:50
jeblairzaro: do you know if there is support for a similar "post this message on push" feature?21:50
jeblairzaro: i'd like to be able to run: git review -m 'This patchset is just a rebase on current master.'21:50
zarojeblair: hmm, not that i have seen.  let me take a look again.21:51
*** smarcet has quit IRC21:51
fungiefoley: you're running it like `tox -r -e py27` right? to make sure you don't have any old pyc cruft in a preexisting tox virtualenv?21:51
*** ddieterly[away] is now known as ddieterly21:51
efoleyfungi: I recreate the environment each time, just to be safe21:52
fungi_ody: i wonder if we have some sort of blackhole situation going on somewhere between you and the git servers, maybe path mtu discovery issues or something which aren't affecting your path to github21:52
*** yamahata has quit IRC21:53
*** jamesmcarthur has joined #openstack-infra21:53
*** yamahata has joined #openstack-infra21:54
*** mriedem has quit IRC21:55
fungi_ody: anyway, i just tested a fresh clone of nova from my home network halfway across the usa and it took less than 60 seconds to complete, but traceroute also says my home isp has a peering relationship with rackspace in equinix near the same data center where the git servers are21:55
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: WIP increase TEMPEST_OS_TEST_TIMEOUT
*** annegentle has quit IRC21:56
fungi_ody: scratch that, i forgot to use traceroute6. i was actually cloning over my gre tunnel from hurricane electric21:56
fungialso seems to peer in the same equinix facility though21:57
*** doug-fish has joined #openstack-infra21:57
_odyI'll look again later, possibly from home after I get done at the gym.  For now I am making progress just using github so I am happy with that.21:58
*** thorst has joined #openstack-infra21:58
fungi_ody: more details will be appreciated when you have any. we definitely want this reachable from everywhere so appreciate the heads up that you're having issues with it21:58
*** burgerk_ has joined #openstack-infra21:58
*** annegentle has joined #openstack-infra21:59
fungiAJaeger: pabelanger: nova other-requirements.txt example seems to be passing all jobs on the latest patchset
*** burgerk has quit IRC22:02
*** Sukhdev has quit IRC22:03
fungiefoley: i'm manually attempting to recreate the failure you're seeing on a held ubuntu trusty node now22:04
pabelangerfungi: excellent22:04
efoleyfungi: Awesome!22:04
*** achanda has joined #openstack-infra22:04
*** fitoduarte has quit IRC22:05
zarojeblair: unfortunately i don't see that feature.  have added a feature request, which i should be able to work on sometime soon.22:05
*** camunoz has joined #openstack-infra22:07
jeblairzaro: neat, that would be cool :)22:08
*** burgerk_ has quit IRC22:08
*** esikachev has quit IRC22:09
*** dizquierdo has quit IRC22:09
*** IlyaG has quit IRC22:10
*** apoorvad has quit IRC22:11
*** tonytan4ever has quit IRC22:14
fungiefoley: manual run failed collectd_ceilometer.tests.test_config.TestConfig.test_missing_value raising AssertionError: error(u'Configuration parameter %s not set.', u'OS_AUTH_URL') call not found22:14
fungiefoley: is that one of the failures you've been encountering?22:14
*** dimtruck is now known as zz_dimtruck22:15
fungiefoley: ran again and got an AssertionError: 10 != 1 in collectd_ceilometer.tests.test_config.TestConfig.test_invalid_value which does look like one of the errors i saw in your logs22:16
fungiefoley: this is with the tip of the collectd-ceilometer-plugin master branch, so definitely looks unrelated to your patch22:16
*** cloudtrainme has quit IRC22:16
*** LaT0rtue has quit IRC22:16
*** tpsilva has quit IRC22:17
*** IlyaG has joined #openstack-infra22:17
zarofungi: when you ran flush-cache to attempt to fix ec2-api project do you remember if you flush all caches?22:18
fungizaro: i used "gerrit flush-caches"22:19
openstackgerritNikhil Komawar proposed openstack-infra/irc-meetings: Change Quota WG time
*** kzaitsev_mb has quit IRC22:20
fungizaro: i did not add --all22:20
*** jamesmcarthur has quit IRC22:21
*** amrith is now known as _amrith_22:21
*** thorst has quit IRC22:21
fungizaro: though seems to imply they're synonymous22:21
*** ddieterly is now known as ddieterly[away]22:23
fungiefoley: i ran again and this time it passed. i suspect you have ordering/isolation issues between your tests and depending on the PYTHONHASHSEED value it gets exposed in different ways (or not at all)22:23
*** thorst has joined #openstack-infra22:24
zarofungi: yep, code also says default is --all22:25
*** Sukhdev has joined #openstack-infra22:25
efoleyfungi: thanks. How do I go about fixing this?22:25
fungiefoley: tox sets a random hash seed, so my guess is either you're running this with fewer worker threads or getting lucky and hitting an ordering where it's happening in just the right order to be able to succeed22:26
fungiefoley: does the "worker balance" output at the end of your local runs indicate 4 worker threads?22:26
*** IlyaG has quit IRC22:27
zarofungi: the error doesn't seem like its the same as the diff timeout issue.  i'm guessing it's probably a timing issue that caused this problem way before we saw the MissingObjectException.22:27
*** ddieterly[away] is now known as ddieterly22:28
efoleyfungi: It uses 24 workers. So this is a paralellism issue22:28
zaroi'm just gonna post the error on Gerrit ML to see if anybody else has seen this issue.22:28
*** ddieterly has quit IRC22:29
*** rockyg has quit IRC22:29
*** markvoelker has quit IRC22:29
fungiefoley: yes, i'm betting that almost all of the tests are running in a separate worker thread for you then, as you have 24 threads for 25 tests as opposed to in the gate we only run 4 threads in parallel and so some of the jobs are getting serialized22:30
fungizaro: thanks for looking22:30
fungizaro: i have a feeling it's due to some sort of momentary race between a change to the repo on disk and gerrit reading something from it, so may be very hard to reproduce22:31
*** harlowja has joined #openstack-infra22:32
*** harlowja has quit IRC22:32
*** thorst has quit IRC22:32
fungiefoley: er, i mean some of the tests are getting serialized22:33
*** harlowja has joined #openstack-infra22:33
*** doug-fish has quit IRC22:37
*** gordc has quit IRC22:43
openstackgerritMatthew Treinish proposed openstack/requirements: Cap Routes < 2.3
*** zeih has joined #openstack-infra22:44
*** sputnik13 has joined #openstack-infra22:48
*** fawadkhaliq has quit IRC22:48
*** fawadkhaliq has joined #openstack-infra22:49
*** zeih has quit IRC22:49
*** esker has joined #openstack-infra22:49
*** piet has quit IRC22:50
*** maishsk has quit IRC22:51
*** sdague has quit IRC22:53
*** harlowja has quit IRC22:54
*** harlowja has joined #openstack-infra22:54
*** tongli has quit IRC22:56
*** david-lyle_ is now known as david-lyle22:57
*** eil397 has left #openstack-infra22:58
*** IlyaG has joined #openstack-infra22:58
*** jamesmcarthur has joined #openstack-infra22:58
*** maishsk has joined #openstack-infra22:59
*** annegentle has quit IRC22:59
ianw2016-03-28 21:13:15.430 | 2016-03-26 16:59 <- timestamp | image build date.  presumably images aren't rolling out to rax?  mostly i'm wondering if running on performance node22:59
ianwhmm, actually we cat cpuinfo somewhere22:59
*** rbrndt_ has quit IRC22:59
*** akshai has quit IRC23:00
*** bpokorny_ has joined #openstack-infra23:02
*** bpokorny_ has quit IRC23:02
*** esker has quit IRC23:02
*** bpokorny_ has joined #openstack-infra23:03
*** esker has joined #openstack-infra23:03
*** ybathia has quit IRC23:04
*** efoley_ has joined #openstack-infra23:04
*** bpokorny has quit IRC23:04
*** tphummel has quit IRC23:05
*** jamesmcarthur has quit IRC23:05
*** tphummel has joined #openstack-infra23:05
*** tonytan4ever has joined #openstack-infra23:05
*** efoley has quit IRC23:05
jrollare there docs on building a bare-trusty image (nearly) identical to infra's?23:07
*** vgridnev has quit IRC23:07
jrollpointer to the code is an acceptable answer here, to23:07
*** LaT0rtue has joined #openstack-infra23:08
*** hichihara has joined #openstack-infra23:08
*** mriedem has joined #openstack-infra23:08
clarkbjroll: we stopped using bare-trusty so no :) the project-config repo has a tools/build-image script that needs updating to use the ubuntu minimal elements23:08
*** sputnik13 has quit IRC23:08
jrollclarkb: oh, what's the new name?23:09
clarkbjroll: if you s/ubuntu/ubuntu-minimal/ and add in the simple-init element that will get you ehat we use23:09
clarkbjroll: just 'ubuntu-trusty'23:09
clarkbcommon image for all your ubuntu needs23:09
fungijroll: yeah, we're now running devstack-based jobs and all other jobs on the same images rather than having one image for devstack and another for everything else23:10
fungimakes things _much_ simpler23:10
jrollclarkb: so tools/build-image, DISTRO=ubuntu-minimal, add simple-init, add whatever element drops in ssh keys, upload to my provider, done?23:10
fungijust took a long time to lay all the groundwork23:10
*** esker has quit IRC23:10
jrollfungi: I'm well aware, been debugging them on and off since the switch, just didn't know the name :)23:10
clarkbjroll: yup be warned its a pretty big image23:10
*** bpokorny_ has quit IRC23:11
ianwhmm, tempest runs +300 more jobs on centos7-minimal images than !minimal images23:11
*** maishsk has quit IRC23:11
fungiianw: s/jobs/tests/? if so, awesome?23:11
*** yuanying has joined #openstack-infra23:11
*** apoorvad has joined #openstack-infra23:11
jrollclarkb: thanks man23:11
ianwsorry, yeah, tests.  not sure if it's a bug or a feature yet23:11
fungiianw: hopefully it's not missing something that's supposed to be making it skip tests on that platform23:12
ianwyeah, or we've inadvertently enabled something23:12
*** ybathia has joined #openstack-infra23:13
*** apoorvad has quit IRC23:13
*** apoorvad has joined #openstack-infra23:13
ianwbut fedora 23 seems to run about 50 minutes on the performance nodes, which is about exactly what i would expect ... about 5 minutes longer than ubuntu & can be attributed to pip cache23:14
*** esker has joined #openstack-infra23:16
*** kzaitsev_mb has joined #openstack-infra23:17
ianwoh, it's neutron23:18
*** annegentle has joined #openstack-infra23:19
*** annegentle has quit IRC23:19
fungioh? what about neutron specifically?23:19
clarkbneutron changes the test count23:20
clarkband runtime they both go up23:20
*** thorst has joined #openstack-infra23:20
fungioh, this was comparing tempest neutron and !neutron jobs?23:20
fungiif so, right not a surprise at all23:20
*** esker has quit IRC23:20
*** maishsk has joined #openstack-infra23:20
*** salv-orlando has quit IRC23:21
*** kzaitsev_mb has quit IRC23:22
*** amitgandhinz has quit IRC23:22
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Update for new images, add EXTRA_ELEMENTS
jrollclarkb: hope that's helpful ^23:22
*** asettle has quit IRC23:24
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add E-R query for bug 1312016
openstackbug 1312016 in neutron "nova libvirtError: Unable to add bridge brqxxx-xx port tapxxx-xx: Device or resource busy" [Medium,Fix released] - Assigned to Andreas Scheuring (andreas-scheuring)23:24
*** harlowja has quit IRC23:25
*** harlowja has joined #openstack-infra23:25
pvaneckHello infra/puppet core. Looking to get merged. It has two +2s, just need the Workflow+1 :)23:27
*** Qiming has joined #openstack-infra23:29
*** mylu has joined #openstack-infra23:31
_odyfungi: Did a quick mtr of my connection to,
openstackgerritMerged openstack-infra/elastic-recheck: Add E-R query for bug 1312016
openstackbug 1312016 in neutron "nova libvirtError: Unable to add bridge brqxxx-xx port tapxxx-xx: Device or resource busy" [Medium,Fix released] - Assigned to Andreas Scheuring (andreas-scheuring)23:35
fungipvaneck: looks like jeblair was holding off approving that waiting for your response, so he may want to approve it23:35
_odyInteresting though, items with packet loss there doesn't have packet loss when going over a comcast link.23:35
*** sputnik13 has joined #openstack-infra23:35
*** esker has joined #openstack-infra23:36
jrolljust realized most of this image will likely be the plethora of openstack repos it is caching, heh23:36
fungijroll: yes, that is precisely most of it23:36
* jroll leaves that out23:37
pvaneckfungi: yea, i'll wait for him23:37
ianwhmm, no smoking gun -> .  will get some more data points23:37
fungi_ody: the "loss" on the interim hops looks like it's probably just throttling icmp responses or filtering the probes rather than responding when their ttl is exceeded. since there's no packet loss to git.o.o according to mtr that's probably not indicative of any issue23:39
*** gildub has joined #openstack-infra23:39
pleia2fungi: it only took all day (ok, I didn't work on it all day) but it turns out their openstackid URL changed, current one in zanata is giving a 404, once I have the right one I should be able to fix it up \o/23:40
fungipleia2: what did it change from/to?23:40
*** achanda has quit IRC23:41
fungirather, what is zanata trying and failing to use?23:41
*** LaT0rtue has quit IRC23:41
pleia2fungi: following up with the user to figure out what their current real one is23:41
fungigot it23:41
pleia2trying to use which doesn't exist for some reason23:41
* pleia2 shrugs23:41
fungiso not (necessarily) an issue with openstackid.org23:41
pleia2no, maybe they changed their name at some point23:41
pleia2so it just doesn't match anymore23:42
fungiyeah, that's been a concern of mine ever since i first saw that openstackid was relying on human names for the openids23:42
funginames change often, you need something immutable23:42
*** knikolla has quit IRC23:44
*** zz_dimtruck is now known as dimtruck23:44
pleia2especially if you're like me (I am better at computers than people)23:45
fungi_ody: i don't seem to be able to reach so can't easily construct the return route to your system from the git server. is there an ip address on the machine you're testing from, to which i can try to traceroute?23:45
*** NobodyCam has quit IRC23:46
fungithough i do hope more of your systems are named after bleach characters23:47
_odyheh.  No just my laptop23:47
_odyOur outbound NAT IP is
*** tonytan4ever has quit IRC23:47
fungicool, i'll trace back to that23:48
fungi_ody: surprisingly, the return route goes through comcast23:48
fungiso this may in fact be related to what bkero mentioned earlier23:49
*** sputnik13 has quit IRC23:49
fungiyour packets are reaching rackspace via hurricane electric, and then rackspace is choosing to send the replies back via comcast23:49
_odyheh.  Good to know.23:50
*** nelsnelson has joined #openstack-infra23:50
*** fawadkhaliq has quit IRC23:50
fungii mean, route selection changes minute to minute so who knows what it'll be tomorrow. bgp is a fickle mistress23:50
*** fawadkhaliq has joined #openstack-infra23:50
_odyWe went ahead and raised a support request with rax.23:51
*** mylu has quit IRC23:51
*** nelsnels_ has quit IRC23:51
fungibut at least some of the internet thinks your network is best reached through comcast23:51
*** davideagnello has quit IRC23:51
fungiand spectrumnet23:51
fungiand opusnet23:51
*** fawadkhaliq has quit IRC23:52
*** fawadkhaliq has joined #openstack-infra23:52
fungifwiw my traceroute from home to you (charter) takes hurricane electric23:53
_odyopusnet is our "other" provider which we do advertise on so that seem normal for part of the internet.23:54
*** LaT0rtue has joined #openstack-infra23:54
*** EricGonczer_ has joined #openstack-infra23:54
*** sigmavirus24 is now known as sigmavirus24_awa23:58
openstackgerritMerged openstack-infra/irc-meetings: Change Quota WG time
*** ajmiller has quit IRC23:58
*** IlyaG has quit IRC23:59
*** fawadkhaliq has quit IRC23:59
*** fawadkhaliq has joined #openstack-infra23:59

Generated by 2.14.0 by Marius Gedminas - find it at!