Saturday, 2016-07-30

*** xarses has quit IRC00:02
*** dimtruck is now known as zz_dimtruck00:02
*** tlbr_ has quit IRC00:02
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Include subnodes in updateStats function  https://review.openstack.org/34911900:04
*** piet has quit IRC00:08
*** jcoufal has joined #openstack-infra00:10
pabelangerclarkb: I think that will render our stats properly now in grafana^.  Waiting for dsvm to run so I can look at statsd output00:10
*** thorst has joined #openstack-infra00:11
*** mtanino has quit IRC00:11
*** jcoufal_ has quit IRC00:12
*** yamamoto has joined #openstack-infra00:13
corvuspabelanger: i think it has a problem00:16
pabelangercorvus: just seen your comments00:17
pabelangerI'll give that a go00:17
*** thorst has quit IRC00:18
*** tqtran_ has quit IRC00:19
*** yamamoto has quit IRC00:19
*** jcoufal_ has joined #openstack-infra00:19
*** jcoufal has quit IRC00:20
*** thorst has joined #openstack-infra00:22
*** kzaitsev_mb has joined #openstack-infra00:23
*** tlbr_ has joined #openstack-infra00:24
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Include subnodes in updateStats function  https://review.openstack.org/34911900:24
pabelangercorvus: I believe that is what you were suggesting ^00:24
*** kzaitsev_mb has quit IRC00:28
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Include subnodes in updateStats function  https://review.openstack.org/34911900:28
*** tlbr_ has quit IRC00:28
*** nwkarsten has quit IRC00:29
*** adrian_otto has quit IRC00:32
*** amitgandhinz has joined #openstack-infra00:35
*** daemontool_ has joined #openstack-infra00:37
*** sputnik13 has quit IRC00:38
*** tkelsey has joined #openstack-infra00:39
*** amitgandhinz has quit IRC00:40
*** sarob has joined #openstack-infra00:40
*** daemontool has quit IRC00:41
*** hurgleburgler has quit IRC00:43
*** tkelsey has quit IRC00:43
pabelangercorvus: still around?00:46
pabelangercorvus: http://grafana.openstack.org/dashboard/db/nodepool looks a little odd ATM00:46
pabelangerwe have 202+ ready nodes00:46
pabelangerI wonder if something is wrong with gearman00:46
*** Guest62949 has joined #openstack-infra00:49
*** Guest62949 has quit IRC00:50
*** jcoufal has joined #openstack-infra00:50
*** jcoufal_ has quit IRC00:50
pabelangerinfra-root: I believe we are having an issue with gearman, see ^00:52
pabelangernot sure if anybody else is around on the friday00:53
*** kzaitsev_mb has joined #openstack-infra00:54
clarkbI am not really around but did a launcher stop working right?00:56
pabelangerclarkb: launchers still appear to be online00:57
clarkbare all those ready nodes predominantly assigned to one?00:59
pabelangernope, even across all launchers01:00
clarkbhuh01:01
pabelangerya, I'm not sure what is going on.  nodepool is still setting some of the nodes to USED01:02
pabelangermaybe a blip in the network01:04
pabelangerokay, things are workingish for now01:04
*** achanda has quit IRC01:05
pabelangerHmm, they are all ubuntu-trusty nodes01:06
*** achanda has joined #openstack-infra01:07
*** hwcomcn has joined #openstack-infra01:11
*** liusheng has quit IRC01:14
openstackgerritMerged openstack-infra/puppet-mosquitto: Add basic permissions and security  https://review.openstack.org/34792301:14
*** yamamoto has joined #openstack-infra01:15
*** tonytan4ever has quit IRC01:18
*** jcoufal_ has joined #openstack-infra01:19
*** jcoufal has quit IRC01:20
*** yamamoto has quit IRC01:21
*** baoli has joined #openstack-infra01:21
*** baoli has quit IRC01:21
*** hieulq_ has joined #openstack-infra01:23
*** kzaitsev_mb has quit IRC01:23
*** daemontool_ has quit IRC01:25
*** jcoufal has joined #openstack-infra01:28
*** jcoufal_ has quit IRC01:28
*** woodster_ has quit IRC01:29
*** mtanino has joined #openstack-infra01:30
*** sarob has quit IRC01:31
*** sarob has joined #openstack-infra01:32
*** rfolco has joined #openstack-infra01:32
*** rfolco has quit IRC01:33
*** ddieterly has joined #openstack-infra01:35
*** amitgandhinz has joined #openstack-infra01:36
*** sputnik13 has joined #openstack-infra01:39
*** amitgandhinz has quit IRC01:41
*** asettle has joined #openstack-infra01:42
*** EricGonczer_ has joined #openstack-infra01:45
*** asettle has quit IRC01:49
*** jcoufal has quit IRC01:51
*** zhurong has joined #openstack-infra01:51
*** sarob has quit IRC01:52
*** jcoufal has joined #openstack-infra01:52
*** zz_dimtruck is now known as dimtruck01:53
*** mtanino has quit IRC01:54
*** tonytan4ever has joined #openstack-infra01:55
*** baoli has joined #openstack-infra01:57
*** itisha has quit IRC02:00
*** baoli has quit IRC02:01
*** dimtruck is now known as zz_dimtruck02:03
*** tlbr_ has joined #openstack-infra02:10
*** jcoufal_ has joined #openstack-infra02:11
*** jcoufal has quit IRC02:11
*** pt_15 has quit IRC02:16
*** yamamoto has joined #openstack-infra02:17
*** tlbr_ has quit IRC02:20
*** kzaitsev_mb has joined #openstack-infra02:20
*** thorst has quit IRC02:22
*** thorst has joined #openstack-infra02:22
*** yamamoto has quit IRC02:22
*** chlong has quit IRC02:23
*** tlbr_ has joined #openstack-infra02:23
*** zhurong has quit IRC02:26
*** coolsvap has joined #openstack-infra02:26
*** xarses has joined #openstack-infra02:27
*** zz_dimtruck is now known as dimtruck02:27
*** tlbr_ has quit IRC02:27
*** thorst has quit IRC02:31
*** sdake has joined #openstack-infra02:34
*** amitgandhinz has joined #openstack-infra02:37
*** tlbr_ has joined #openstack-infra02:40
*** tkelsey has joined #openstack-infra02:41
*** ddieterly has quit IRC02:41
*** amitgandhinz has quit IRC02:43
*** dimtruck is now known as zz_dimtruck02:44
*** tlbr_ has quit IRC02:44
*** tkelsey has quit IRC02:45
*** jcoufal_ has quit IRC02:47
*** jcoufal has joined #openstack-infra02:48
*** tlbr_ has joined #openstack-infra02:49
*** jcoufal has quit IRC02:50
*** tlbr_ has quit IRC02:57
*** kzaitsev_mb has quit IRC02:57
openstackgerritMerged openstack-infra/puppet-mosquitto: Add websocket listner to mosquitto  https://review.openstack.org/34792403:02
*** links has joined #openstack-infra03:02
*** gouthamr has quit IRC03:03
*** julim has quit IRC03:08
*** nwkarsten has joined #openstack-infra03:09
*** sxmxc has joined #openstack-infra03:11
*** nwkarsten has quit IRC03:13
*** Liuqing has joined #openstack-infra03:15
*** tqtran has joined #openstack-infra03:17
*** sdake has quit IRC03:18
*** yamamoto has joined #openstack-infra03:19
openstackgerritMerged openstack-infra/puppet-mosquitto: Remove trailing whitespace from mosquitto.conf template  https://review.openstack.org/34830703:19
*** tqtran has quit IRC03:23
*** kzaitsev_mb has joined #openstack-infra03:23
*** piet has joined #openstack-infra03:24
*** yamamoto has quit IRC03:25
*** achanda has quit IRC03:25
*** sdake has joined #openstack-infra03:29
*** thorst has joined #openstack-infra03:29
*** kzaitsev_mb has quit IRC03:31
*** tlbr_ has joined #openstack-infra03:32
*** thorst has quit IRC03:36
*** tlbr_ has quit IRC03:36
*** hieulq_ has quit IRC03:38
*** amitgandhinz has joined #openstack-infra03:39
*** hurgleburgler has joined #openstack-infra03:41
*** hurgleburgler has quit IRC03:42
*** piet has quit IRC03:43
*** amitgandhinz has quit IRC03:43
*** yamahata has quit IRC03:44
*** HeOS has quit IRC03:48
*** sxmxc has quit IRC03:48
*** EricGonczer_ has quit IRC03:50
*** sdake has quit IRC03:57
*** rossella_s has quit IRC04:00
*** rossella_s has joined #openstack-infra04:00
*** sdake has joined #openstack-infra04:01
*** sdake has quit IRC04:01
*** Guest48593 has joined #openstack-infra04:02
*** ramishra has quit IRC04:02
*** zz_dimtruck is now known as dimtruck04:02
*** sabari has left #openstack-infra04:04
*** asettle has joined #openstack-infra04:06
*** Guest48593 has quit IRC04:07
*** tlbr_ has joined #openstack-infra04:07
*** piet has joined #openstack-infra04:10
*** asettle has quit IRC04:11
*** tlbr_ has quit IRC04:13
*** achanda has joined #openstack-infra04:13
*** yamamoto has joined #openstack-infra04:20
*** Liuqing has quit IRC04:21
*** dimtruck is now known as zz_dimtruck04:26
*** yamamoto has quit IRC04:26
*** kzaitsev_mb has joined #openstack-infra04:27
*** kzaitsev_mb has quit IRC04:32
*** thorst has joined #openstack-infra04:34
*** snarwade has quit IRC04:37
*** amitgandhinz has joined #openstack-infra04:40
*** thorst has quit IRC04:41
*** amitgandhinz has quit IRC04:44
*** rbuzatu has joined #openstack-infra04:49
*** yamahata has joined #openstack-infra04:50
*** tlbr_ has joined #openstack-infra04:50
*** tlbr_ has quit IRC04:51
*** psachin has joined #openstack-infra04:55
*** bhunter71 has quit IRC04:55
*** rbuzatu has quit IRC04:57
pabelanger#status log zuul-launcher release ran on zl04-zl07, I've left the first 4 zuul-launchers so we can debug the "too many ready node online" issue05:00
openstackstatuspabelanger: finished logging05:00
*** snarwade has joined #openstack-infra05:01
*** zhurong has joined #openstack-infra05:03
*** tlbr_ has joined #openstack-infra05:12
*** tlbr_ has quit IRC05:16
*** siva_krish has joined #openstack-infra05:17
*** zhurong has quit IRC05:17
*** zhurong has joined #openstack-infra05:21
*** piet has quit IRC05:22
*** yamamoto has joined #openstack-infra05:22
*** amitkqed has quit IRC05:26
*** amitkqed has joined #openstack-infra05:26
*** ociuhandu has quit IRC05:27
*** yamamoto has quit IRC05:28
*** kzaitsev_mb has joined #openstack-infra05:28
*** siva_krish has quit IRC05:33
*** zhurong has quit IRC05:34
*** kzaitsev_mb has quit IRC05:35
*** zhurong has joined #openstack-infra05:35
*** tlbr_ has joined #openstack-infra05:38
*** thorst has joined #openstack-infra05:39
*** amitgandhinz has joined #openstack-infra05:40
*** tlbr_ has quit IRC05:42
*** adrian_otto has joined #openstack-infra05:43
*** psilvad has quit IRC05:43
*** tkelsey has joined #openstack-infra05:45
*** amitgandhinz has quit IRC05:45
*** thorst has quit IRC05:46
*** tkelsey has quit IRC05:47
*** yamamoto has joined #openstack-infra05:48
*** tlbr_ has joined #openstack-infra05:56
*** tlbr_ has quit IRC05:57
*** tonytan4ever has quit IRC05:57
*** tlbr_ has joined #openstack-infra05:57
*** yamamoto has quit IRC05:59
*** tlbr_ has quit IRC06:02
*** Guest15615 has joined #openstack-infra06:10
*** Guest15615 has quit IRC06:15
*** lucas_ai has joined #openstack-infra06:15
*** tqtran has joined #openstack-infra06:20
*** lucas_ai has quit IRC06:20
*** hwcomcn has quit IRC06:22
*** tqtran has quit IRC06:24
*** kzaitsev_mb has joined #openstack-infra06:30
*** lucas_ai has joined #openstack-infra06:30
*** Liuqing has joined #openstack-infra06:35
*** abregman has joined #openstack-infra06:36
*** kzaitsev_mb has quit IRC06:38
*** Liuqing has quit IRC06:41
*** amitgandhinz has joined #openstack-infra06:41
*** yamahata has quit IRC06:41
*** tlbr_ has joined #openstack-infra06:41
*** hieulq_ has joined #openstack-infra06:42
*** Liuqing has joined #openstack-infra06:42
*** thorst has joined #openstack-infra06:44
*** amitgandhinz has quit IRC06:46
*** tlbr_ has quit IRC06:46
*** sputnik13 has quit IRC06:51
*** thorst has quit IRC06:51
*** hieulq_ has quit IRC06:57
*** tonytan4ever has joined #openstack-infra07:00
*** yamamoto has joined #openstack-infra07:00
*** tonytan4ever has quit IRC07:05
*** lucas_ai has quit IRC07:05
*** yamamoto has quit IRC07:06
*** tlbr_ has joined #openstack-infra07:10
*** tlbr_ has quit IRC07:15
*** kzaitsev_mb has joined #openstack-infra07:17
*** Liuqing has quit IRC07:24
*** chem` has joined #openstack-infra07:26
*** chem has quit IRC07:28
*** apetrich has quit IRC07:29
*** tlbr_ has joined #openstack-infra07:29
*** tlbr_ has quit IRC07:33
*** sputnik13 has joined #openstack-infra07:37
*** armax has joined #openstack-infra07:39
*** armax has quit IRC07:40
*** tlbr_ has joined #openstack-infra07:41
*** amitgandhinz has joined #openstack-infra07:42
*** tkelsey has joined #openstack-infra07:44
*** e0ne has joined #openstack-infra07:45
*** tlbr_ has quit IRC07:45
*** amitgandhinz has quit IRC07:46
*** adrian_otto has quit IRC07:46
*** thorst has joined #openstack-infra07:49
*** tkelsey has quit IRC07:49
*** rdx_matt has joined #openstack-infra07:49
*** kzaitsev_mb has quit IRC07:50
*** kzaitsev_mb has joined #openstack-infra07:52
*** e0ne has quit IRC07:53
*** hieulq_ has joined #openstack-infra07:55
*** thorst has quit IRC07:56
*** apetrich has joined #openstack-infra07:57
*** kzaitsev_mb has quit IRC07:57
*** tlbr_ has joined #openstack-infra07:58
openstackgerritMerged openstack-infra/project-config: Cross test constraints and project unittests.  https://review.openstack.org/30873907:59
*** rossella_s has quit IRC07:59
*** zzzeek has quit IRC08:00
*** rossella_s has joined #openstack-infra08:00
*** rdx_matt has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** asettle has joined #openstack-infra08:00
*** yamamoto has joined #openstack-infra08:02
*** adrian_otto has joined #openstack-infra08:04
*** rdx_matt has joined #openstack-infra08:04
*** tlbr_ has quit IRC08:05
*** adrian_otto1 has joined #openstack-infra08:06
*** yamamoto has quit IRC08:08
*** adrian_otto has quit IRC08:09
*** claudiub|2 has joined #openstack-infra08:09
*** rdx_matt has quit IRC08:12
*** hieulq_ has quit IRC08:17
*** adrian_otto1 has quit IRC08:18
*** hieulq_ has joined #openstack-infra08:19
*** tqtran has joined #openstack-infra08:21
*** Guest62208 has joined #openstack-infra08:22
*** adrian_otto has joined #openstack-infra08:24
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add initial err2d2 project config  https://review.openstack.org/34899808:25
*** tqtran has quit IRC08:26
*** Guest62208 has quit IRC08:26
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add initial err2d2 project config  https://review.openstack.org/34899808:28
AJaegerpabelanger: Something strange with grafana: http://grafana.openstack.org/dashboard/db/nodepool gives "Timeseries data request errors"08:33
*** adrian_otto1 has joined #openstack-infra08:33
*** adrian_otto has quit IRC08:33
*** kzaitsev_mb has joined #openstack-infra08:34
*** adrian_otto1 has quit IRC08:39
*** achanda has quit IRC08:39
*** adrian_otto has joined #openstack-infra08:42
*** tlbr_ has joined #openstack-infra08:43
*** amitgandhinz has joined #openstack-infra08:43
*** amitgandhinz has quit IRC08:47
*** tlbr_ has quit IRC08:48
AJaegerinfra-root, https://review.openstack.org/348998 just had three post failures ;(08:49
*** adrian_otto has quit IRC08:50
*** r-mibu has quit IRC08:51
*** r-mibu has joined #openstack-infra08:51
*** adrian_otto has joined #openstack-infra08:53
*** thorst has joined #openstack-infra08:55
*** snarwade has quit IRC08:59
*** adrian_otto has quit IRC09:01
*** thorst has quit IRC09:01
*** adrian_otto has joined #openstack-infra09:01
*** yamamoto has joined #openstack-infra09:04
*** hieulq_ has quit IRC09:04
*** kzaitsev_mb has quit IRC09:06
*** ianychoi has quit IRC09:06
*** tldr__ has quit IRC09:07
*** yamamoto has quit IRC09:10
*** tlbr_ has joined #openstack-infra09:16
*** ianychoi has joined #openstack-infra09:18
*** tlbr_ has quit IRC09:19
*** yamahata has joined #openstack-infra09:24
*** cgalan has quit IRC09:25
*** elo has quit IRC09:26
*** nherciu has joined #openstack-infra09:28
*** nwkarsten has joined #openstack-infra09:30
*** nherciu has quit IRC09:30
*** elo has joined #openstack-infra09:30
*** tlbr_ has joined #openstack-infra09:32
*** daemontool_ has joined #openstack-infra09:32
AJaegerinfra-root, I see POST_FAILURES everywhere on zuul ;(09:34
*** nwkarsten has quit IRC09:35
*** hieulq_ has joined #openstack-infra09:35
*** tlbr_ has quit IRC09:36
*** achanda has joined #openstack-infra09:39
*** hieulq_ has quit IRC09:40
*** adrian_otto has quit IRC09:41
*** daemontool_ has quit IRC09:43
*** amitgandhinz has joined #openstack-infra09:44
*** achanda has quit IRC09:47
*** amitgandhinz has quit IRC09:48
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Enable constraints for releasenotes  https://review.openstack.org/34918109:49
*** asettle has quit IRC09:52
*** thorst has joined #openstack-infra09:58
AJaeger#status log All jobs currenty fail with POST_FAILURE10:00
openstackstatusAJaeger: finished logging10:00
*** daemontool_ has joined #openstack-infra10:03
*** bhavik has joined #openstack-infra10:04
*** thorst has quit IRC10:06
*** yamamoto has joined #openstack-infra10:06
*** yamamoto has quit IRC10:11
*** bhavik has quit IRC10:25
*** yamahata has quit IRC10:26
yolandaAJaeger, static disk is full, i'm taking a look10:27
*** yamamoto has joined #openstack-infra10:30
yolandaAJaeger, the file /var/lib/mlocate in static has 32g10:31
yolandait's exhausting disk space under root10:31
*** yamamoto has quit IRC10:34
*** zhurong has quit IRC10:35
*** bhavik has joined #openstack-infra10:44
*** achanda has joined #openstack-infra10:44
*** amitgandhinz has joined #openstack-infra10:44
*** yamamoto has joined #openstack-infra10:47
*** achanda has quit IRC10:48
*** amitgandhinz has quit IRC10:49
*** yamamoto has quit IRC10:49
*** dkehn_ has quit IRC10:50
*** dkehn has quit IRC10:50
*** bhavik has quit IRC10:50
*** pt_15 has joined #openstack-infra10:50
*** daemontool_ has quit IRC10:52
*** tlbr_ has joined #openstack-infra10:53
yolandainfra-root, /srv/static/logs is failing, but i need to go for an appointment now, can you take a look when you come?10:53
*** dkehn has joined #openstack-infra10:56
*** tlbr_ has quit IRC10:58
*** tonytan4ever has joined #openstack-infra11:02
*** dkehn_ has joined #openstack-infra11:02
*** thorst has joined #openstack-infra11:05
*** tonytan4ever has quit IRC11:07
*** claudiub|2 has quit IRC11:09
*** thorst has quit IRC11:11
*** tkelsey has joined #openstack-infra11:28
*** yamamoto has joined #openstack-infra11:30
*** bhavik has joined #openstack-infra11:30
*** tkelsey has quit IRC11:35
*** daemontool_ has joined #openstack-infra11:40
*** yamamoto has quit IRC11:43
*** yamamoto has joined #openstack-infra11:44
*** yamamoto has quit IRC11:44
*** amitgandhinz has joined #openstack-infra11:45
*** zhurong has joined #openstack-infra11:49
*** yamamoto has joined #openstack-infra11:49
*** amitgandhinz has quit IRC11:50
*** pt_15 has quit IRC11:52
*** pt_15 has joined #openstack-infra11:53
*** jak-jak has quit IRC11:56
*** rossella_s has quit IRC12:00
*** rossella_s has joined #openstack-infra12:00
*** daemontool_ has quit IRC12:02
*** jak-jak has joined #openstack-infra12:05
*** jak-jak is now known as Guest9390012:06
*** thorst has joined #openstack-infra12:11
*** EricGonczer_ has joined #openstack-infra12:14
*** thorst has quit IRC12:16
*** thorst has joined #openstack-infra12:20
*** tlbr_ has joined #openstack-infra12:23
*** tqtran has joined #openstack-infra12:25
*** links has quit IRC12:28
*** thorst has quit IRC12:28
*** tlbr_ has quit IRC12:29
*** tqtran has quit IRC12:29
*** Na3iL has joined #openstack-infra12:39
*** clenimar has quit IRC12:42
*** ericksonsantos has quit IRC12:45
*** ericksonsantos has joined #openstack-infra12:46
*** amitgandhinz has joined #openstack-infra12:46
*** achanda has joined #openstack-infra12:46
*** amitgandhinz has quit IRC12:50
*** achanda has quit IRC12:51
*** clenimar has joined #openstack-infra12:54
*** sarob has joined #openstack-infra12:57
*** sarob has quit IRC13:02
*** dgonzalez has quit IRC13:03
*** itisha has joined #openstack-infra13:04
*** dgonzalez has joined #openstack-infra13:11
*** yamamoto has quit IRC13:13
*** mriedem has joined #openstack-infra13:22
mordredkrotscheck: I was afk - but the location isn't exported because /etc/openstack/clouds.yaml is the canonical global location for that file - you should only need to specify location if it's somehwere other than /etc/openstack/clouds.yaml or ~/.config/openstack/clouds.yaml13:23
mordredyolanda: disk on static seems fine now? did you fix it?13:24
*** markvoelker has joined #openstack-infra13:25
*** thorst has joined #openstack-infra13:29
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add updatedb.conf file for static.openstack.org  https://review.openstack.org/34919413:32
*** markvoelker has quit IRC13:32
*** ddieterly has joined #openstack-infra13:32
*** cgalan has joined #openstack-infra13:35
*** thorst has quit IRC13:37
*** rkukura has quit IRC13:38
*** zhurong has quit IRC13:38
AJaegermordred: doesn't seem to be fixed, see your POST_FAILUREs ^13:39
mordredlovely13:43
AJaegeryeah ;(13:44
*** Jeffrey4l_ has joined #openstack-infra13:46
*** amitgandhinz has joined #openstack-infra13:47
mordredwell, there's this:[2469503.045379] init: Failed to write to log file /var/log/upstart/systemd-logind.log13:47
*** tonytan4ever has joined #openstack-infra13:47
mordredbut also a ton of io errors13:47
*** Jeffrey4l has quit IRC13:47
mordredlooking like something went poorly with a disk or sometihng?13:47
*** ddieterly is now known as ddieterly[away]13:47
*** tlbr_ has joined #openstack-infra13:48
*** ddieterly[away] is now known as ddieterly13:50
*** amitgandhinz has quit IRC13:51
*** tlbr_ has quit IRC13:52
*** sarob has joined #openstack-infra13:54
*** sarob has quit IRC13:58
*** yamamoto has joined #openstack-infra14:07
fungiyolanda: yes, rackspace warned us that they were going to interrupt access to cinder this morning. rcarrillocruz planned to be around to handle it14:07
fungii'm guessing that hasn't happened yet, but i'll see if i can identify which volumes are now offline14:08
*** ddieterly has quit IRC14:09
fungifrom syslog it appears that the logs, docs-draft and tarballs volumes have been switched to read-only14:11
fungiyolanda: mordred: i see you're both also logged in. are you repairing those or should i be working on that now?14:11
mordredfungi: I am not repairing - I'm going to afk in a few minutes and didn't want to tie anything up14:12
mordred(we're visiting nola this weekend)14:12
fungienjoy!14:12
fungii'm temporarily stopping puppet for static.o.o14:15
fungii've commented out the ssh key for the jenkins account on it now so that uploads can't happen while i have volumes unmounted14:16
fungiand stopped apache so that it holds no open file handles for the various docroots14:17
fungi#status alert Cinder connectivity was lost to the volumes for sites served from static.openstack.org (logs, docs-draft, tarballs) and so they will remain offline until repairs are complete14:19
openstackstatusfungi: sending alert14:19
*** piet has joined #openstack-infra14:19
*** coolsvap has quit IRC14:21
-openstackstatus- NOTICE: Cinder connectivity was lost to the volumes for sites served from static.openstack.org (logs, docs-draft, tarballs) and so they will remain offline until repairs are complete14:22
*** ChanServ changes topic to "Cinder connectivity was lost to the volumes for sites served from static.openstack.org (logs, docs-draft, tarballs) and so they will remain offline until repairs are complete"14:22
fungiyolanda: i suspect that was your root session i just killed. its cwd was one of the volumes i was trying to umount14:23
fungiokay, i've temporarily disabled the sites for those three volumes and started apache again so that the other sites will still be served14:24
openstackstatusfungi: finished sending alert14:25
fungitarballs.o.o is cleaned up and served in apache again14:26
*** bhavik has quit IRC14:26
*** tqtran has joined #openstack-infra14:26
fungiinfra-root: i have fsck runs going for docs-draft and logs in parallel in different windows of a root screen session on static.o.o14:27
*** markvoelker has joined #openstack-infra14:28
*** tlbr has joined #openstack-infra14:30
*** tqtran has quit IRC14:31
*** tkelsey has joined #openstack-infra14:31
*** tlbr is now known as Guest3732714:31
*** yamamoto has quit IRC14:31
*** yamamoto has joined #openstack-infra14:32
*** markvoelker has quit IRC14:34
*** Guest37327 has quit IRC14:35
*** tkelsey has quit IRC14:37
*** thorst has joined #openstack-infra14:38
*** jcoufal has joined #openstack-infra14:40
*** thorst has quit IRC14:43
*** EricGonczer_ has quit IRC14:43
*** tlbr_ has joined #openstack-infra14:44
*** tlbr_ has quit IRC14:45
*** jcoufal has quit IRC14:45
*** markvoelker has joined #openstack-infra14:47
*** amitgandhinz has joined #openstack-infra14:47
*** jcoufal has joined #openstack-infra14:48
*** achanda has joined #openstack-infra14:49
*** markvoelker has quit IRC14:51
*** jcoufal has quit IRC14:51
*** amitgandhinz has quit IRC14:52
*** zhurong has joined #openstack-infra14:54
*** jcoufal has joined #openstack-infra14:54
*** achanda has quit IRC14:54
yolandamordred, fungi, i did not fix it. I had to run out for a family lunch14:56
*** Guest32207 has joined #openstack-infra15:07
*** abregman has quit IRC15:13
*** Guest32207 has quit IRC15:14
*** sdake has joined #openstack-infra15:19
fungidocs-draft is back online15:24
fungilogs is, as expected, still fsck'ing15:24
fungii expect it to be a few hours still15:24
fungigonna catch up on some yardwork but will check in on it periodically and wrap up when it finishes15:25
*** rfolco has joined #openstack-infra15:30
*** flepied1 has quit IRC15:31
*** ekarlso has quit IRC15:34
AJaegerthanks, fungi!15:35
* AJaeger is finished now with garden work ;)15:36
*** siva_krish has joined #openstack-infra15:37
rcarrillocruzSorry folks, I got the virus from my son and wife, 39 C fever15:39
rcarrillocruzBack to bed, hope all is good15:40
*** thorst has joined #openstack-infra15:41
*** rbuzatu has joined #openstack-infra15:41
*** sdake has quit IRC15:42
*** piet has quit IRC15:44
*** zhurong has quit IRC15:46
*** psachin has quit IRC15:47
*** zhurong has joined #openstack-infra15:47
*** markvoelker has joined #openstack-infra15:47
*** amitgandhinz has joined #openstack-infra15:48
*** thorst has quit IRC15:49
*** zhurong has quit IRC15:49
*** rfolco has quit IRC15:50
*** markvoelker has quit IRC15:53
*** amitgandhinz has quit IRC15:53
*** rbuzatu has quit IRC16:00
AJaegerrcarrillocruz, all the best for you and your family!16:00
*** rbuzatu has joined #openstack-infra16:01
pabelangerAJaeger: I think it is possible that graphite.o.o is having the same volume issues as static.o.o, I have not checked yet however16:09
*** ekarlso has joined #openstack-infra16:09
pabelangerAJaeger: Yup, graphite.openstack.org/main01 is not attached. I believe fungi is working with it, so not going to do anything at the moment16:10
*** Guest30537 has joined #openstack-infra16:11
fungipabelanger: the maintenance ticket rackspace opened a couple weeks ago didn't mention any of graphite's volumes were going to be impacted, just static.openstack.org/main0416:14
fungii was unaware any other servers were affected, but maybe i misread (or maybe they screwed up the maintenance)16:14
*** Guest30537 has quit IRC16:15
pabelangerfungi: I could be wrong, let see dive deeper16:15
pabelangerfungi: actually, looks like graphite.o.o has 2 volumes, main01 is not attached16:16
pabelangerlet me see what that is used for16:16
fungii expect that was leftover from a migration to larger volumes or something16:17
fungiregardless, there's a lot of errors in dmesg/syslog for the /var/lib/graphite/storage filesystem16:18
fungiwe probably need to umount, fsck and mount it16:18
pabelangerhttp://paste.openstack.org/show/544367/16:18
pabelangeris pvs and vgs16:19
fungiyeah, i'm logged into it too16:19
pabelangerfungi: do you mind adding that into your list? I have to run some errands shortly here16:19
fungiyeah, i'll knock it out16:19
*** Jeffrey4l_ has quit IRC16:20
pabelangerThanks, I'll look at your history when I get back and add some docs to http://docs.openstack.org/infra/system-config/sysadmin.html#cinder-volume-management16:20
*** Jeffrey4l_ has joined #openstack-infra16:21
fungii have apache stopped on the server, that fs unmounted and a fsck going in a root screen session16:21
fungiooh, logs fsck is nearly wrapped up16:22
*** lucas-dinner has quit IRC16:22
fungiit's up to Pass 4: Checking reference counts16:22
pabelangergreat16:24
fungiokay, graphite is fixed and back online16:25
fungifsck ran clean16:25
*** kzaitsev_mb has joined #openstack-infra16:26
fungilogs is still on Pass 5: Checking group summary information16:27
*** tqtran has joined #openstack-infra16:28
*** lucasagomes has joined #openstack-infra16:28
fungii need to go get back to my yardwork but will check in on it again in a bit16:28
pabelangersame, stepping away for a bit too16:28
pabelangerwill check back in a few16:28
*** tqtran has quit IRC16:32
*** yamamoto_ has joined #openstack-infra16:40
*** yamamoto has quit IRC16:44
fungilogs just finished--bringing it back online now16:44
*** thorst has joined #openstack-infra16:46
*** bhavik has joined #openstack-infra16:47
*** e0ne has joined #openstack-infra16:48
*** markvoelker has joined #openstack-infra16:48
*** amitgandhinz has joined #openstack-infra16:49
fungiugh, just noticed i didn't wait long enough to disable the jenkins ssh authorized key after disabling puppet16:49
fungicleaning that up now16:49
*** kzaitsev_mb has quit IRC16:51
*** achanda has joined #openstack-infra16:52
*** thorst has quit IRC16:52
*** markvoelker has quit IRC16:52
fungi#status ok Emergency filesystem repairs are complete; any changes which failed jobs with POST_FAILURE status or due to lack of access to tarballs can be safely rechecked now16:53
openstackstatusfungi: sending ok16:53
*** amitgandhinz has quit IRC16:53
*** dkehn has quit IRC16:54
*** dkehn_ has quit IRC16:54
*** roxanaghe has joined #openstack-infra16:55
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/"16:55
-openstackstatus- NOTICE: Emergency filesystem repairs are complete; any changes which failed jobs with POST_FAILURE status or due to lack of access to tarballs can be safely rechecked now16:55
*** karthikp_ has joined #openstack-infra16:56
*** achanda has quit IRC16:57
openstackstatusfungi: finished sending ok16:59
*** roxanaghe has quit IRC17:00
*** dkehn has joined #openstack-infra17:01
*** ihrachys has joined #openstack-infra17:04
*** karthikp_ has quit IRC17:05
*** dkehn_ has joined #openstack-infra17:06
*** kzaitsev_mb has joined #openstack-infra17:07
*** esberglu has joined #openstack-infra17:08
*** gouthamr has joined #openstack-infra17:12
*** bhavik has quit IRC17:13
*** esberglu has quit IRC17:13
*** kzaitsev_mb has quit IRC17:15
*** ericksonsantos has quit IRC17:15
*** ericksonsantos has joined #openstack-infra17:17
*** ihrachys has quit IRC17:18
*** ihrachys has joined #openstack-infra17:23
fungilooks like they screwed up the maintenance... i've just acknowledged individual tickets for static.openstack.org/main04 (expected in the maintenance), static.openstack.org/main05 (not anticipated), zuul.openstack.org/debug01 (thankfully not in use!), and graphite.openstack.org/main0117:34
fungiso looks like we've got it all covered at this point17:37
*** nwkarsten has joined #openstack-infra17:41
*** pt_15 has quit IRC17:46
*** bhavik has joined #openstack-infra17:48
*** markvoelker has joined #openstack-infra17:49
*** amitgandhinz has joined #openstack-infra17:50
*** thorst has joined #openstack-infra17:51
*** achanda has joined #openstack-infra17:52
*** markvoelker has quit IRC17:53
*** amitgandhinz has quit IRC17:55
*** thorst has quit IRC17:58
*** e0ne has quit IRC17:59
*** e0ne_ has joined #openstack-infra17:59
*** ddieterly has joined #openstack-infra18:03
*** Na3iL has quit IRC18:07
*** rfolco has joined #openstack-infra18:07
*** e0ne_ has quit IRC18:14
*** baoli has joined #openstack-infra18:15
*** ihrachys_ has joined #openstack-infra18:16
*** aviau has quit IRC18:17
*** aviau has joined #openstack-infra18:17
*** ddieterly has quit IRC18:17
*** ihrachys has quit IRC18:18
*** rfolco has quit IRC18:20
*** e0ne has joined #openstack-infra18:20
*** e0ne has quit IRC18:20
*** karthikp_ has joined #openstack-infra18:25
*** chem` has quit IRC18:25
*** karthik__ has joined #openstack-infra18:27
*** karthikp_ has quit IRC18:29
*** achanda has quit IRC18:31
*** baoli has quit IRC18:36
*** roxanaghe has joined #openstack-infra18:39
*** roxanaghe has quit IRC18:40
*** roxanagh_ has joined #openstack-infra18:41
*** ddieterly has joined #openstack-infra18:43
*** hongbin has joined #openstack-infra18:43
*** markvoelker has joined #openstack-infra18:50
*** amitgandhinz has joined #openstack-infra18:50
*** jcoufal_ has joined #openstack-infra18:51
*** jcoufal has quit IRC18:52
*** oanson has joined #openstack-infra18:52
*** markvoelker has quit IRC18:55
rcarrillocruzAJaeger: thx, fortunately my son and wife are now good, i hope I don't take much to pass it...18:55
*** amitgandhinz has quit IRC18:55
*** thorst has joined #openstack-infra18:55
*** ddieterly has quit IRC18:59
*** nwkarsten has quit IRC19:01
*** roxanagh_ has quit IRC19:02
*** thorst has quit IRC19:03
*** roxanaghe has joined #openstack-infra19:03
rcarrillocruzfungi: checking my mail from my phone it seems the infracloud servers got iLO ips assigned19:06
rcarrillocruzi'll check them out when my brain is back to useful19:07
*** fifieldt has quit IRC19:07
rcarrillocruzthey updated the ticket friday late on my tz19:07
*** tlbr has joined #openstack-infra19:09
*** sdake has joined #openstack-infra19:13
*** tlbr has quit IRC19:13
*** jcoufal_ has quit IRC19:16
*** fifieldt has joined #openstack-infra19:19
*** jcoufal has joined #openstack-infra19:19
*** sdake has quit IRC19:20
*** Guest83635 has joined #openstack-infra19:23
*** EricGonczer_ has joined #openstack-infra19:26
*** Guest83635 has quit IRC19:27
*** weshay_afk has quit IRC19:28
*** sdake has joined #openstack-infra19:28
*** oanson has quit IRC19:30
*** jcoufal has quit IRC19:32
*** openstackgerrit_ has joined #openstack-infra19:34
*** vgridnev has quit IRC19:36
*** openstackgerrit_ has quit IRC19:36
*** _degorenko|afk has quit IRC19:36
*** jcoufal has joined #openstack-infra19:38
*** kzaitsev_mb has joined #openstack-infra19:39
*** siva_krish has quit IRC19:40
sdakeanyyeone know what htis is aout: [12:36:22] openstackgerrit_ (~openstack@176.74.218.106) left IRC (K-Lined)19:40
*** yamamoto_ has quit IRC19:41
sdakehttps://en.wikipedia.org/wiki/IRCd#K-line19:41
*** asettle has joined #openstack-infra19:42
*** karthik__ has quit IRC19:43
*** asettle has quit IRC19:46
*** karthikp_ has joined #openstack-infra19:47
*** achanda has joined #openstack-infra19:47
*** baoli has joined #openstack-infra19:49
*** baoli has quit IRC19:50
openstackgerritkrozin proposed openstack-infra/project-config: Add openstack/pipe-report project  https://review.openstack.org/33954919:50
*** baoli has joined #openstack-infra19:50
*** markvoelker has joined #openstack-infra19:50
*** achanda has quit IRC19:51
*** amitgandhinz has joined #openstack-infra19:51
fungisdake: my guess is that might be harlowja testing out his errbot-based gerritbot rewrite19:52
*** jcoufal has quit IRC19:52
fungircarrillocruz: awesome--excellent news!19:52
sdakefungi ok - ust seems like a problem that a major bot is klined ;)19:52
sdakekline = freeenode ban19:53
*** karthikp_ has quit IRC19:54
*** baoli has quit IRC19:55
*** markvoelker has quit IRC19:55
*** amitgandhinz has quit IRC19:56
*** asettle has joined #openstack-infra19:58
*** asettle has quit IRC19:58
*** jcoufal has joined #openstack-infra19:58
*** flepied has joined #openstack-infra19:59
*** roxanagh_ has joined #openstack-infra20:00
*** rossella_s has quit IRC20:00
*** ihrachys_ has quit IRC20:00
*** rossella_s has joined #openstack-infra20:00
*** ihrachys has joined #openstack-infra20:00
*** thorst has joined #openstack-infra20:01
*** roxanaghe has quit IRC20:02
*** kzaitsev_mb has quit IRC20:03
*** ihrachys has quit IRC20:04
*** thorst has quit IRC20:07
*** kzaitsev_mb has joined #openstack-infra20:09
*** sdake has quit IRC20:15
*** kzaitsev_mb has quit IRC20:16
*** bhavik has quit IRC20:16
*** jcoufal has quit IRC20:17
*** jcoufal has joined #openstack-infra20:18
*** thorst has joined #openstack-infra20:25
*** thorst has quit IRC20:28
*** sdake has joined #openstack-infra20:28
*** hongbin_ has joined #openstack-infra20:33
*** hongbin has quit IRC20:33
*** hongbin_ has quit IRC20:35
*** Jeffrey4l_ has quit IRC20:35
*** hongbin has joined #openstack-infra20:35
*** Jeffrey4l_ has joined #openstack-infra20:36
*** kzaitsev_mb has joined #openstack-infra20:39
*** yamamoto has joined #openstack-infra20:41
*** jcoufal has quit IRC20:46
*** sdake has quit IRC20:48
*** yamamoto has quit IRC20:48
potsCan anyone point me to instructions for bringing up devstack w/ kilo?  It seems some components (keystone,glance) dont have stable/kilo branches.20:48
*** markvoelker has joined #openstack-infra20:51
*** amitgandhinz has joined #openstack-infra20:52
*** flepied has quit IRC20:54
*** markvoelker has quit IRC20:56
*** amitgandhinz has quit IRC20:56
*** jcoufal has joined #openstack-infra20:58
*** kzaitsev_mb has quit IRC21:01
*** eric_lopez has joined #openstack-infra21:07
*** elo has quit IRC21:09
pabelangerShrews: Any idea why nodepool is failing to start zookeeper for unit testing? http://logs.openstack.org/19/349119/3/check/gate-nodepool-python27-db-ubuntu-xenial/9432bb5/console.html#_2016-07-30_02_19_07_35579121:10
*** thorst has joined #openstack-infra21:14
*** thorst has quit IRC21:14
*** thorst has joined #openstack-infra21:14
*** ianychoi has quit IRC21:17
*** eric_lopez has quit IRC21:19
*** bauwser is now known as bauzas_off21:22
*** thorst has quit IRC21:23
*** elo has joined #openstack-infra21:26
*** achanda has joined #openstack-infra21:29
*** piet has joined #openstack-infra21:30
*** tqtran has joined #openstack-infra21:30
*** hongbin has quit IRC21:33
*** tqtran has quit IRC21:34
*** karthikp_ has joined #openstack-infra21:47
*** roxanagh_ has quit IRC21:52
*** markvoelker has joined #openstack-infra21:52
*** jcoufal has quit IRC21:52
*** amitgandhinz has joined #openstack-infra21:53
fungipots: they have "kilo-eol" tags instead. the branches were tagged and deleted once they passed end of upstream support21:54
*** piet has quit IRC21:57
*** markvoelker has quit IRC21:57
*** amitgandhinz has quit IRC21:57
*** jcoufal has joined #openstack-infra21:58
*** kzaitsev_mb has joined #openstack-infra21:58
*** kzaitsev_mb has quit IRC22:02
*** jcoufal has quit IRC22:02
*** rbuzatu has quit IRC22:08
*** itisha has quit IRC22:10
*** tonytan4ever has quit IRC22:15
*** gouthamr has quit IRC22:15
*** tonytan4ever has joined #openstack-infra22:15
*** jcoufal has joined #openstack-infra22:17
*** thorst has joined #openstack-infra22:21
*** thorst has quit IRC22:28
*** kzaitsev_mb has joined #openstack-infra22:28
*** kzaitsev_mb has quit IRC22:34
*** achanda has quit IRC22:41
*** jcoufal has quit IRC22:52
*** markvoelker has joined #openstack-infra22:53
*** amitgandhinz has joined #openstack-infra22:54
*** itisha has joined #openstack-infra22:54
*** markvoelker has quit IRC22:57
*** jcoufal has joined #openstack-infra22:58
*** amitgandhinz has quit IRC22:58
*** hparekh has quit IRC22:59
*** pt_15 has joined #openstack-infra22:59
pabelangerheh, 1 job in the check queue.23:00
*** thorst has joined #openstack-infra23:06
*** thorst has quit IRC23:08
*** rbuzatu has joined #openstack-infra23:09
*** hparekh has joined #openstack-infra23:11
*** thorst has joined #openstack-infra23:11
*** rbuzatu has quit IRC23:13
*** thorst has quit IRC23:17
*** karthik__ has joined #openstack-infra23:24
*** sdake has joined #openstack-infra23:26
*** karthikp_ has quit IRC23:26
*** kzaitsev_mb has joined #openstack-infra23:29
*** sdake has quit IRC23:32
*** jcoufal has quit IRC23:32
*** kzaitsev_mb has quit IRC23:35
*** jcoufal has joined #openstack-infra23:37
*** asettle has joined #openstack-infra23:39
*** asettle has quit IRC23:44
*** jcoufal has quit IRC23:49
*** markvoelker has joined #openstack-infra23:53
*** amitgandhinz has joined #openstack-infra23:54
*** jcoufal has joined #openstack-infra23:57
*** markvoelker has quit IRC23:58
*** amitgandhinz has quit IRC23:59
*** rossella_s has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!