| *** yamamoto has quit IRC | 00:01 | |
| *** yamamoto has joined #openstack-infra | 00:01 | |
| *** sflanigan has joined #openstack-infra | 00:10 | |
| *** dingyichen has joined #openstack-infra | 00:13 | |
| *** funzo has joined #openstack-infra | 00:13 | |
| *** xarses_ has joined #openstack-infra | 00:14 | |
| *** andreww has joined #openstack-infra | 00:17 | |
| *** spzala has joined #openstack-infra | 00:18 | |
| *** funzo has quit IRC | 00:18 | |
| *** andreww has quit IRC | 00:18 | |
| *** xarses_ has quit IRC | 00:19 | |
| *** andreww has joined #openstack-infra | 00:20 | |
| *** gildub has joined #openstack-infra | 00:21 | |
| *** spzala has quit IRC | 00:22 | |
| *** baoli has joined #openstack-infra | 00:26 | |
| *** andreww has quit IRC | 00:34 | |
| *** aeng has joined #openstack-infra | 00:37 | |
| *** thorst has joined #openstack-infra | 00:38 | |
| *** spzala has joined #openstack-infra | 00:41 | |
| *** thorst has quit IRC | 00:43 | |
| *** LindaWang has joined #openstack-infra | 00:43 | |
| *** seanhandley has joined #openstack-infra | 00:47 | |
| *** yamamoto has quit IRC | 00:52 | |
| *** jamesmcarthur has joined #openstack-infra | 00:53 | |
| *** jamesmcarthur has quit IRC | 00:54 | |
| *** yamamoto has joined #openstack-infra | 00:55 | |
| *** jamesmcarthur has joined #openstack-infra | 00:55 | |
| *** zhurong has joined #openstack-infra | 00:58 | |
| *** funzo has joined #openstack-infra | 01:14 | |
| *** jamesmcarthur has quit IRC | 01:16 | |
| *** funzo has quit IRC | 01:18 | |
| *** jamesmcarthur has joined #openstack-infra | 01:20 | |
| *** mattmceuen has joined #openstack-infra | 01:23 | |
| *** jamesmcarthur has quit IRC | 01:24 | |
| *** jamesmcarthur has joined #openstack-infra | 01:24 | |
| *** yamamoto has quit IRC | 01:26 | |
| *** jamesmcarthur has quit IRC | 01:36 | |
| *** thorst has joined #openstack-infra | 01:39 | |
| *** thorst has quit IRC | 01:43 | |
| *** gildub has quit IRC | 01:44 | |
| *** yamamoto has joined #openstack-infra | 01:48 | |
| *** annegentle has joined #openstack-infra | 01:52 | |
| *** aeng has quit IRC | 01:53 | |
| *** aeng has joined #openstack-infra | 01:54 | |
| *** yamamoto has quit IRC | 01:58 | |
| *** aeng has quit IRC | 01:58 | |
| *** cuongnv has joined #openstack-infra | 01:58 | |
| *** yamamoto has joined #openstack-infra | 01:59 | |
| *** hongbin has joined #openstack-infra | 02:01 | |
| *** gildub has joined #openstack-infra | 02:01 | |
| *** gouthamr has quit IRC | 02:07 | |
| *** yamamoto has quit IRC | 02:10 | |
| *** aeng has joined #openstack-infra | 02:11 | |
| *** annegentle has quit IRC | 02:12 | |
| *** yamamoto has joined #openstack-infra | 02:13 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Install Laravel schedule https://review.openstack.org/493275 | 02:16 |
|---|---|---|
| *** liujiong has joined #openstack-infra | 02:19 | |
| *** abelur_ has joined #openstack-infra | 02:19 | |
| *** gcb has joined #openstack-infra | 02:30 | |
| *** baoli has quit IRC | 02:36 | |
| *** spzala has quit IRC | 02:37 | |
| *** thorst has joined #openstack-infra | 02:40 | |
| *** felipemonteiro has joined #openstack-infra | 02:44 | |
| *** thorst has quit IRC | 02:44 | |
| *** caphrim007 has joined #openstack-infra | 02:47 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/elastic-recheck master: Add query for bug 1710533 https://review.openstack.org/493409 | 02:58 |
| openstack | bug 1710533 in OpenStack-Gate "Job fails to download container from docker.io" [Undecided,New] https://launchpad.net/bugs/1710533 | 02:58 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for bug 1710533 https://review.openstack.org/493409 | 03:05 |
| openstack | bug 1710533 in OpenStack-Gate "Job fails to download container from docker.io" [Undecided,New] https://launchpad.net/bugs/1710533 | 03:05 |
| *** skelso has joined #openstack-infra | 03:10 | |
| *** funzo has joined #openstack-infra | 03:16 | |
| *** funzo has quit IRC | 03:20 | |
| *** skelso has quit IRC | 03:23 | |
| *** psachin has joined #openstack-infra | 03:34 | |
| *** udesale has joined #openstack-infra | 03:37 | |
| *** ykarel has joined #openstack-infra | 03:53 | |
| *** adisky__ has joined #openstack-infra | 03:56 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use prepare command to build container images env file https://review.openstack.org/483178 | 03:59 |
| *** tnarg has joined #openstack-infra | 04:03 | |
| *** armax has joined #openstack-infra | 04:05 | |
| *** darkside101 has joined #openstack-infra | 04:06 | |
| *** links has joined #openstack-infra | 04:07 | |
| *** pgadiya has joined #openstack-infra | 04:13 | |
| *** funzo has joined #openstack-infra | 04:16 | |
| *** ramishra has joined #openstack-infra | 04:17 | |
| *** arturb has joined #openstack-infra | 04:19 | |
| *** hongbin has quit IRC | 04:19 | |
| *** funzo has quit IRC | 04:21 | |
| arturb | hi. Anyone knows how to republish the data in project service-type-authotiry? I am talking about this change https://review.openstack.org/#/c/490750/ | 04:23 |
| *** zhurong has quit IRC | 04:27 | |
| *** zhurong has joined #openstack-infra | 04:29 | |
| *** claudiub has joined #openstack-infra | 04:30 | |
| *** armax has quit IRC | 04:32 | |
| *** felipemonteiro has quit IRC | 04:32 | |
| *** spzala has joined #openstack-infra | 04:38 | |
| *** thorst has joined #openstack-infra | 04:41 | |
| *** spzala has quit IRC | 04:42 | |
| *** thorst has quit IRC | 04:46 | |
| *** gildub has quit IRC | 04:46 | |
| *** Daisy_ has joined #openstack-infra | 04:47 | |
| *** Daisy_ has quit IRC | 04:47 | |
| *** Dinesh_Bhor has joined #openstack-infra | 04:49 | |
| *** sree has joined #openstack-infra | 04:52 | |
| *** liujiong_lj has joined #openstack-infra | 04:58 | |
| *** liujiong has quit IRC | 05:00 | |
| *** ramishra has quit IRC | 05:04 | |
| *** darkside101 has quit IRC | 05:05 | |
| *** ramishra has joined #openstack-infra | 05:06 | |
| *** hyakuhei has quit IRC | 05:12 | |
| *** funzo has joined #openstack-infra | 05:17 | |
| *** funzo has quit IRC | 05:22 | |
| AJaeger | arturb: two options: an admin needs to republish - or somebody merges another change in | 05:30 |
| *** udesale__ has joined #openstack-infra | 05:34 | |
| *** udesale has quit IRC | 05:37 | |
| *** e0ne has joined #openstack-infra | 05:39 | |
| *** e0ne has quit IRC | 05:39 | |
| *** thorst has joined #openstack-infra | 05:42 | |
| *** florianf has joined #openstack-infra | 05:45 | |
| openstackgerrit | Sampath Priyankara (samP) proposed openstack-infra/project-config master: Add pypi-jobs to masakari and related projects https://review.openstack.org/493434 | 05:47 |
| *** thorst has quit IRC | 05:47 | |
| *** florianf has quit IRC | 05:48 | |
| openstackgerrit | Sampath Priyankara (samP) proposed openstack-infra/project-config master: Add pypi-jobs to masakari and related projects https://review.openstack.org/493434 | 05:49 |
| *** links has quit IRC | 05:50 | |
| *** florianf has joined #openstack-infra | 05:51 | |
| *** links has joined #openstack-infra | 05:56 | |
| *** udesale__ has quit IRC | 06:16 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/glean master: ignore ip6gre0 when setting up interfaces https://review.openstack.org/493443 | 06:16 |
| *** jbadiapa has joined #openstack-infra | 06:17 | |
| *** eranrom has quit IRC | 06:19 | |
| *** yamamoto has quit IRC | 06:21 | |
| *** liujiong_lj is now known as liujiong | 06:26 | |
| *** ramishra has quit IRC | 06:27 | |
| *** udesale__ has joined #openstack-infra | 06:27 | |
| *** dhajare has joined #openstack-infra | 06:28 | |
| *** rwsu has joined #openstack-infra | 06:32 | |
| *** thorst has joined #openstack-infra | 06:43 | |
| *** ramishra has joined #openstack-infra | 06:45 | |
| *** dhajare has quit IRC | 06:46 | |
| *** dhajare has joined #openstack-infra | 06:47 | |
| *** thorst has quit IRC | 06:48 | |
| *** rcernin has joined #openstack-infra | 06:56 | |
| *** ykarel has quit IRC | 06:58 | |
| *** ykarel has joined #openstack-infra | 07:03 | |
| *** martinkopec has joined #openstack-infra | 07:05 | |
| *** camunoz has joined #openstack-infra | 07:07 | |
| *** shardy has joined #openstack-infra | 07:09 | |
| *** tesseract has joined #openstack-infra | 07:10 | |
| *** ramishra has quit IRC | 07:11 | |
| *** ramishra has joined #openstack-infra | 07:12 | |
| *** funzo has joined #openstack-infra | 07:19 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 07:22 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Refactor provider config to driver module https://review.openstack.org/488384 | 07:22 |
| *** aarefiev_afk is now known as aarefiev | 07:22 | |
| *** jpena|off is now known as jpena | 07:22 | |
| *** funzo has quit IRC | 07:23 | |
| *** ccamacho has joined #openstack-infra | 07:30 | |
| *** jtomasek has joined #openstack-infra | 07:32 | |
| *** jpich has joined #openstack-infra | 07:35 | |
| *** spzala has joined #openstack-infra | 07:38 | |
| *** spzala has quit IRC | 07:39 | |
| *** spzala has joined #openstack-infra | 07:39 | |
| *** egonzalez has joined #openstack-infra | 07:39 | |
| *** spzala has quit IRC | 07:39 | |
| *** AJaeger has quit IRC | 07:41 | |
| *** thorst has joined #openstack-infra | 07:44 | |
| *** sree has quit IRC | 07:47 | |
| *** thorst has quit IRC | 07:49 | |
| *** AJaeger has joined #openstack-infra | 07:49 | |
| *** yamamoto has joined #openstack-infra | 07:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: add stable-compat jobs for tempest and devstack-gate for pike https://review.openstack.org/493098 | 07:56 |
| openstackgerrit | Merged openstack-infra/project-config master: add pike bitrot jobs https://review.openstack.org/493099 | 08:01 |
| openstackgerrit | Merged openstack-infra/project-config master: add periodic pike jobs for tempest https://review.openstack.org/493100 | 08:01 |
| openstackgerrit | Merged openstack-infra/project-config master: Add nuetron-dynamic-routing to Dragonflow's tempest's local.conf https://review.openstack.org/492626 | 08:05 |
| *** dalvarez has quit IRC | 08:05 | |
| *** dmellado has quit IRC | 08:06 | |
| *** ltomasbo has quit IRC | 08:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: networking-midonet: remove v2 grenade job from master https://review.openstack.org/493091 | 08:06 |
| *** e0ne has joined #openstack-infra | 08:07 | |
| openstackgerrit | Merged openstack-infra/project-config master: Skip long-running jobs for only placement-api-ref changes https://review.openstack.org/493242 | 08:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove UUID token from Patrole jobs https://review.openstack.org/493118 | 08:09 |
| openstackgerrit | Merged openstack-infra/project-config master: zuul: layout: osa-os_sahara: Add nv centos job https://review.openstack.org/492450 | 08:10 |
| openstackgerrit | Merged openstack-infra/project-config master: Dragonflow: increase timeout for fullstack jobs https://review.openstack.org/492416 | 08:10 |
| *** flaper87 has quit IRC | 08:10 | |
| *** xarses has quit IRC | 08:12 | |
| *** yamamoto has quit IRC | 08:12 | |
| openstackgerrit | Merged openstack-infra/project-config master: Bringing upstream training virtual environment over here https://review.openstack.org/490202 | 08:13 |
| *** xarses has joined #openstack-infra | 08:13 | |
| *** yamamoto has joined #openstack-infra | 08:14 | |
| *** flaper87 has joined #openstack-infra | 08:16 | |
| *** flaper87 has quit IRC | 08:18 | |
| *** flaper87 has joined #openstack-infra | 08:18 | |
| *** funzo has joined #openstack-infra | 08:19 | |
| *** derekh has joined #openstack-infra | 08:22 | |
| *** yamamoto has quit IRC | 08:23 | |
| *** ralonsoh has joined #openstack-infra | 08:23 | |
| *** funzo has quit IRC | 08:24 | |
| *** ykarel is now known as ykarel|lunch | 08:24 | |
| *** Guest4459 has quit IRC | 08:25 | |
| *** lucasagomes has joined #openstack-infra | 08:35 | |
| *** lucasagomes has quit IRC | 08:36 | |
| *** lucas-afk is now known as lucasagomes | 08:37 | |
| *** tosky has joined #openstack-infra | 08:37 | |
| *** kjackal_ has joined #openstack-infra | 08:38 | |
| *** jaosorior has joined #openstack-infra | 08:41 | |
| *** yamamoto has joined #openstack-infra | 08:43 | |
| *** sree has joined #openstack-infra | 08:47 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** florianf_ has joined #openstack-infra | 08:54 | |
| *** florianf has quit IRC | 08:54 | |
| *** dingyichen has quit IRC | 09:04 | |
| *** ramishra has quit IRC | 09:10 | |
| *** sambetts|afk is now known as sambetts | 09:10 | |
| *** ramishra has joined #openstack-infra | 09:11 | |
| *** electrofelix has joined #openstack-infra | 09:11 | |
| *** ykarel|lunch is now known as ykarel | 09:14 | |
| *** sshnaidm|off has quit IRC | 09:15 | |
| *** funzo has joined #openstack-infra | 09:20 | |
| *** kjackal_ has quit IRC | 09:22 | |
| *** funzo has quit IRC | 09:25 | |
| *** openstackgerrit has joined #openstack-infra | 09:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/493494 | 09:33 |
| *** kornicameister has quit IRC | 09:33 | |
| *** ykarel is now known as ykarel|afk | 09:36 | |
| *** spzala has joined #openstack-infra | 09:40 | |
| *** ralonsoh has quit IRC | 09:43 | |
| *** tushar has joined #openstack-infra | 09:44 | |
| *** spzala has quit IRC | 09:44 | |
| tushar | hi All , Facing issue while accessing the nodepool nodes in thirdparty CI setup | 09:45 |
| *** thorst has joined #openstack-infra | 09:45 | |
| tushar | Is there any change in DIB image ? for which we need to changes our setup accordingly? | 09:46 |
| AJaeger | tushar: so, that'S your nodepool nodes not ours, correct? | 09:46 |
| AJaeger | tushar: I'm not aware of significant changes - but please double check the commit history of the repos. | 09:47 |
| tushar | yes , nodes in our setup | 09:47 |
| tushar | AJaeger : ok | 09:47 |
| AJaeger | tushar: for third party CI, there's #openstack-third-party-ci channel, best to ask other operators there. | 09:47 |
| *** Hal has joined #openstack-infra | 09:48 | |
| *** Hal is now known as Guest78801 | 09:48 | |
| AJaeger | tushar: and if that does not help, I suggest to ask later here again (during US time) | 09:49 |
| *** shardy has quit IRC | 09:50 | |
| *** thorst has quit IRC | 09:50 | |
| *** shardy has joined #openstack-infra | 09:51 | |
| *** cuongnv has quit IRC | 09:51 | |
| tushar | Ajaeger : yes sure | 09:53 |
| tushar | thank you | 09:53 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add kpartx as a requirement to build images https://review.openstack.org/493004 | 09:54 |
| hwoarang | pabelanger dirk: the removal of opensuse422 jobs broke a lot of stuff for me :( | 09:54 |
| hwoarang | 42.2 is still alive and 42.3 is unstable so now I am left with 0 working jobs again | 09:55 |
| * hwoarang is sad | 09:55 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Clear up debian element documentation https://review.openstack.org/491971 | 09:55 |
| dirk | hwoarang: more details? | 09:56 |
| dirk | hwoarang: why is 42.3 unstable? it got released as latest stable opensuse release.. | 09:56 |
| hwoarang | https://bugzilla.opensuse.org/show_bug.cgi?id=1053595 | 09:56 |
| openstack | bugzilla.opensuse.org bug 1053595 in Basesystem "Could not set limit: mkfs.btrfs died abnormally" [Normal,New] - Assigned to bnc-team-screening | 09:56 |
| hwoarang | but why remove 42.2 when it's still supported? please give me some time to stabilize 42.3 since it was added to the CI less than a month ago :( | 09:57 |
| hwoarang | at least in openstack ansible projects | 09:57 |
| *** efoley has joined #openstack-infra | 09:58 | |
| *** LindaWang has quit IRC | 09:58 | |
| *** efoley_ has joined #openstack-infra | 09:59 | |
| *** efoley has quit IRC | 10:02 | |
| hwoarang | can we please revert that commit until the end of month so I can work on solving the 42.3 issues whilst keeping 42.2 jobs to a green state? | 10:03 |
| hwoarang | oh looks like 422 images have been removed as well | 10:05 |
| *** sree has quit IRC | 10:06 | |
| *** sshnaidm|off has joined #openstack-infra | 10:07 | |
| *** sshnaidm|off is now known as sshnaidm | 10:08 | |
| dirk | hwoarang: i asked the same, pabelanger mentioned hr wants to do the same like with fedora where only one version exists in infra | 10:09 |
| dirk | hwoarang: happy to help on getting 42.3 to work though | 10:10 |
| hwoarang | ok i guess it does not worth the effort to get 422 back | 10:10 |
| *** Guest78801 has quit IRC | 10:11 | |
| *** liujiong has quit IRC | 10:11 | |
| *** efoley__ has joined #openstack-infra | 10:27 | |
| *** efoley_ has quit IRC | 10:27 | |
| *** efoley_ has joined #openstack-infra | 10:28 | |
| *** efoley__ has quit IRC | 10:28 | |
| *** efoley_ has quit IRC | 10:31 | |
| *** efoley_ has joined #openstack-infra | 10:31 | |
| *** dtantsur|afk is now known as dtantsur | 10:32 | |
| *** udesale__ has quit IRC | 10:33 | |
| *** florianf_ is now known as florianf | 10:36 | |
| *** yamamoto has quit IRC | 10:37 | |
| *** baoli has joined #openstack-infra | 10:37 | |
| *** yamamoto has joined #openstack-infra | 10:40 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** baoli has quit IRC | 10:41 | |
| *** seanhandley has quit IRC | 10:42 | |
| *** thorst has joined #openstack-infra | 10:46 | |
| yuval | Hey infra, seems like there are a lot of queued jobs (some are queued for over an hour) | 10:46 |
| *** Hal has joined #openstack-infra | 10:47 | |
| yuval | any idea? | 10:47 |
| *** Hal is now known as Guest31152 | 10:47 | |
| AJaeger | yuval: we have less capacity - and all nodes are in use. Check the graph on http://status.openstack.org/zuul/ | 10:48 |
| yuval | AJaeger: :\ | 10:48 |
| *** ykarel|afk is now known as ykarel | 10:48 | |
| AJaeger | yuval: so, please have patience | 10:49 |
| *** jkilpatr has quit IRC | 10:49 | |
| *** jkilpatr has joined #openstack-infra | 10:49 | |
| *** thorst has quit IRC | 10:51 | |
| ykarel | AJaeger, hi | 10:51 |
| ykarel | AJaeger, i have some doubt for sphinx warning-is-error, should i ask here | 10:52 |
| *** isaacb has joined #openstack-infra | 10:52 | |
| AJaeger | ykarel: #openstack-doc might be better | 10:54 |
| ykarel | AJaeger, Ok, i was looking on #openstack-docs :( | 10:55 |
| *** yamamoto has joined #openstack-infra | 10:59 | |
| *** yamamoto has quit IRC | 11:04 | |
| *** mat128 has joined #openstack-infra | 11:08 | |
| *** lucasagomes is now known as lucas-hungry | 11:15 | |
| *** shardy is now known as shardy_lunch | 11:15 | |
| *** seanhandley_ has joined #openstack-infra | 11:17 | |
| *** hwoarang has quit IRC | 11:18 | |
| *** hwoarang has joined #openstack-infra | 11:19 | |
| *** funzo has joined #openstack-infra | 11:22 | |
| *** funzo has quit IRC | 11:26 | |
| *** yamamoto has joined #openstack-infra | 11:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/493494 | 11:32 |
| *** jpena is now known as jpena|lunch | 11:38 | |
| *** sshnaidm has quit IRC | 11:41 | |
| *** HeOS has joined #openstack-infra | 11:43 | |
| *** sshnaidm|mtg has joined #openstack-infra | 11:46 | |
| *** rlandy has joined #openstack-infra | 11:46 | |
| *** gcb has quit IRC | 11:52 | |
| *** dprince has joined #openstack-infra | 11:56 | |
| *** nicolasbock has joined #openstack-infra | 11:57 | |
| *** frickler has quit IRC | 11:57 | |
| *** shardy_lunch is now known as shardy | 12:01 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/elastic-recheck master: Track intermittent mirrors issues for jobs in e-r https://review.openstack.org/493520 | 12:03 |
| *** slagle has joined #openstack-infra | 12:04 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/elastic-recheck master: Track intermittent mirrors issues for jobs in e-r https://review.openstack.org/493520 | 12:05 |
| *** sree has joined #openstack-infra | 12:07 | |
| dirk | hwoarang: I'm playing with https://review.openstack.org/#/c/493522 | 12:08 |
| *** psachin has quit IRC | 12:08 | |
| dirk | I bet there is somewhere a patch that assumes that btrsprogs is always available | 12:08 |
| *** SergK has quit IRC | 12:09 | |
| *** eranrom has joined #openstack-infra | 12:09 | |
| *** psachin has joined #openstack-infra | 12:10 | |
| hwoarang | dirk: we maintain a global bindep.txt in the openstack-ansible-tests repository | 12:11 |
| hwoarang | but lets see if the tests succeed | 12:11 |
| *** sree has quit IRC | 12:11 | |
| *** Julien-zte has quit IRC | 12:13 | |
| *** pgadiya has quit IRC | 12:13 | |
| *** thorst has joined #openstack-infra | 12:14 | |
| *** xinliang has quit IRC | 12:14 | |
| *** Julien-zte has joined #openstack-infra | 12:15 | |
| *** jianghuaw has quit IRC | 12:15 | |
| *** trown|outtypewww is now known as trown | 12:18 | |
| *** eranrom has quit IRC | 12:20 | |
| *** lucas-hungry is now known as lucasagomes | 12:21 | |
| *** funzo has joined #openstack-infra | 12:23 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/elastic-recheck master: Track intermittent connection issues for centos cloud images https://review.openstack.org/493525 | 12:23 |
| *** sree has joined #openstack-infra | 12:25 | |
| *** jpena|lunch is now known as jpena | 12:25 | |
| *** bogdando has joined #openstack-infra | 12:25 | |
| *** zhurong has quit IRC | 12:25 | |
| *** ldnunes has joined #openstack-infra | 12:26 | |
| bogdando | folks, PTAL these improvements to uncategorised job failures https://review.openstack.org/#/c/493525/ https://review.openstack.org/#/c/493520/ | 12:26 |
| *** funzo has quit IRC | 12:27 | |
| *** isaacb has quit IRC | 12:28 | |
| *** jangutter has joined #openstack-infra | 12:29 | |
| *** isaacb has joined #openstack-infra | 12:31 | |
| *** ldnunes has quit IRC | 12:35 | |
| *** isaacb has quit IRC | 12:36 | |
| *** krtaylor has quit IRC | 12:37 | |
| *** frickler has joined #openstack-infra | 12:37 | |
| *** frickler has quit IRC | 12:37 | |
| *** frickler has joined #openstack-infra | 12:37 | |
| *** spzala has joined #openstack-infra | 12:40 | |
| *** spzala has quit IRC | 12:41 | |
| *** spzala has joined #openstack-infra | 12:41 | |
| *** spzala has quit IRC | 12:41 | |
| *** krtaylor has joined #openstack-infra | 12:41 | |
| *** isaacb has joined #openstack-infra | 12:41 | |
| *** dave-mccowan has joined #openstack-infra | 12:44 | |
| *** ldnunes has joined #openstack-infra | 12:47 | |
| *** markvoelker has joined #openstack-infra | 12:48 | |
| *** sshnaidm|mtg has quit IRC | 12:52 | |
| *** ldnunes has quit IRC | 12:53 | |
| *** dhajare has quit IRC | 12:57 | |
| *** mat128 has quit IRC | 12:57 | |
| *** fried_rice is now known as efried | 12:58 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/elastic-recheck master: Track intermittent connection issues (very slow transfer rates) https://review.openstack.org/493535 | 12:58 |
| *** mat128 has joined #openstack-infra | 12:59 | |
| bogdando | one more ^^ | 13:00 |
| *** jamesmcarthur has joined #openstack-infra | 13:01 | |
| *** mat128 has quit IRC | 13:01 | |
| *** nicolasbock has quit IRC | 13:03 | |
| *** ldnunes has joined #openstack-infra | 13:05 | |
| *** psachin has quit IRC | 13:06 | |
| *** jcoufal has joined #openstack-infra | 13:06 | |
| *** sshnaidm|mtg has joined #openstack-infra | 13:07 | |
| *** trown is now known as trown|brb | 13:07 | |
| *** spzala has joined #openstack-infra | 13:08 | |
| *** baoli has joined #openstack-infra | 13:10 | |
| *** ldnunes has quit IRC | 13:13 | |
| *** claudiub|2 has joined #openstack-infra | 13:18 | |
| *** claudiub has quit IRC | 13:21 | |
| fungi | infra-root: over the weekend ovh support triaged that week-old trouble ticket, and indicated that the reason we haven't heard back from jean-daniel is that he was on vacation both last week and (probably) this week as well | 13:22 |
| *** sshnaidm|mtg is now known as sshnaidm | 13:22 | |
| fungi | infra-root: also, probably good idea to go ahead with https://review.openstack.org/493120 so we can get back a couple hundred nodes capacity | 13:23 |
| *** mattmceuen has quit IRC | 13:23 | |
| *** funzo has joined #openstack-infra | 13:23 | |
| *** aviau has quit IRC | 13:24 | |
| *** aviau has joined #openstack-infra | 13:24 | |
| *** trown|brb is now known as trown | 13:25 | |
| *** ldnunes has joined #openstack-infra | 13:26 | |
| mordred | fungi: done | 13:26 |
| fungi | mordred: thanks, i'll try to keep an eye on the mirror load there and stay on the lookout for any further failures local to rax.ord | 13:26 |
| *** LindaWang has joined #openstack-infra | 13:27 | |
| mordred | fungi: cool - so, also for reasons I don't understand, zuul is ignoring this: https://review.openstack.org/#/c/493365 ... config looks good to me though | 13:28 |
| *** funzo has quit IRC | 13:28 | |
| AJaeger | mordred: I commented in the broken change, let me find it again for you | 13:28 |
| fungi | dependency loop? | 13:28 |
| AJaeger | https://review.openstack.org/#/c/462140/ - dependency loop | 13:29 |
| AJaeger | mordred: ^ | 13:29 |
| *** LindaWang has quit IRC | 13:32 | |
| *** ldnunes has quit IRC | 13:33 | |
| *** gcb has joined #openstack-infra | 13:33 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 13:33 |
| pabelanger | dirk: hwoarang: Ya, for opensuse we want to keep only the latest release of an image in nodepool, to help reduce the amount of images we upload to clouds. Because opensuse doesn't have an LTS, it falls into the same process for fedora, only keep latest release online. | 13:33 |
| AJaeger | could any project-config core review https://review.openstack.org/493080 to update translations for pike, please? | 13:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Temporarily stop using rax-ord" https://review.openstack.org/493120 | 13:35 |
| *** jamesden_ has joined #openstack-infra | 13:37 | |
| *** jamesdenton has quit IRC | 13:38 | |
| *** felipemonteiro has joined #openstack-infra | 13:39 | |
| *** LindaWang has joined #openstack-infra | 13:43 | |
| *** EricGonczer_ has joined #openstack-infra | 13:44 | |
| *** hongbin has joined #openstack-infra | 13:44 | |
| *** felipemonteiro has quit IRC | 13:45 | |
| pabelanger | fungi: do we still want to slowly roll https://review.openstack.org/493112/ into production? clarkb says old cache dirs still get used | 13:45 |
| *** eharney has joined #openstack-infra | 13:45 | |
| fungi | pabelanger: nope, seems perfectly safe to just approve and let it take effect as it wants | 13:45 |
| fungi | thanks! | 13:45 |
| fungi | are you available to keep an eye on it? i didn't want to approve until more people were awake | 13:46 |
| pabelanger | Ya, ready to support now | 13:46 |
| *** ldnunes has joined #openstack-infra | 13:46 | |
| fungi | in it goes | 13:46 |
| pabelanger | yay | 13:46 |
| tosky | hi, this translation-import job for Sahara failed with a strange error (missing msgmerge): http://logs.openstack.org/12/492912/2/check/gate-sahara-releasenotes/b7995a6/console.html | 13:47 |
| *** links has quit IRC | 13:47 | |
| tosky | did something change on the images? | 13:47 |
| *** lbragstad has joined #openstack-infra | 13:48 | |
| pabelanger | fungi: clarkb: I am also wondering if we should experiment with a larger flavor in citycloud-kna1 today too. Seeing a larger amount of failures with jobs failing to download from the reverse proxy cache | 13:48 |
| *** claudiub has joined #openstack-infra | 13:48 | |
| mordred | AJaeger: oh thnak you! I suck :) | 13:49 |
| AJaeger | mordred: no, you don't ;) | 13:49 |
| pabelanger | tosky: not sure, maybe need to add msgmerge to bindep? cc AJaeger | 13:49 |
| mordred | AJaeger: wait - https://review.openstack.org/#/c/493361 isn't testing either ... | 13:50 |
| mordred | AJaeger: and that's at the base of the chain? | 13:50 |
| *** ihrachys has quit IRC | 13:51 | |
| tosky | uhm, release notes translations | 13:51 |
| *** ihrachys has joined #openstack-infra | 13:51 | |
| fungi | pabelanger: i do see some gaps in cacti graphs for that mirror, but those seem to coincide with spikes in cpu activity not network traffic | 13:52 |
| *** claudiub|2 has quit IRC | 13:52 | |
| *** jamesmcarthur has quit IRC | 13:52 | |
| pabelanger | fungi: Agree. Right now we only have 2C/4GB flavors. Was thinking trying with 4C? | 13:52 |
| hwoarang | pabelanger: ok no problem then. i was just expecting a bit more overlap between 422 and 423 but it was a bad timinig since i was away for 2 weeks :) | 13:53 |
| *** jamesmcarthur has joined #openstack-infra | 13:53 | |
| fungi | mordred: all open changes involved in a dependency loop fail to enqueue because zuul can't untangle the complete set into an acyclic graph | 13:53 |
| tosky | uhm, looks like a bindep indeed missing | 13:53 |
| *** jamesmcarthur has quit IRC | 13:53 | |
| *** jamesmcarthur has joined #openstack-infra | 13:53 | |
| tosky | we did not spot the missing bindep on gettext only because we didn't have releasenotes translations | 13:53 |
| openstackgerrit | Merged openstack-infra/system-config master: Make apache cache structure saner https://review.openstack.org/493112 | 13:54 |
| *** mattmceuen has joined #openstack-infra | 13:54 | |
| pabelanger | hwoarang: Ya, moving forward I think it will be smoother. Since opensuse-422 was our first attempt at opensuse, it had some missing devstack dependencies I think. I believe dirk managed to land them in opensuse-423. So, we should get 6 /12 months (I don't actually know when next release happens) this time | 13:54 |
| fungi | pabelanger: looks like cacti isn't finding oids for disk i/o activity, but everything (aside from what's in afs) is on the rootfs | 13:56 |
| mhayden | ianw: thanks for taking a gander at the CentOS repo review ;) | 13:56 |
| pabelanger | fungi: I am not sure why that is | 13:57 |
| fungi | actually, of the two graph gaps in the past 24 hours, one likes up with a bried period where the cpu was maxed out ~75% system and 25% user, the other with a load average spike of an order of magnitude from its average levels | 13:58 |
| *** martinkopec has quit IRC | 13:58 | |
| fungi | so in this case, a flavor with more cpu might help? | 13:58 |
| fungi | tough to say | 13:58 |
| fungi | s/likes/lines/ s/bried/brief/ | 13:58 |
| fungi | oh, we also don't have any swap space provisioned there, i guess because of the lack of auxiliary block devices | 13:59 |
| *** isaacb has quit IRC | 13:59 | |
| mordred | fungi: ah. neat. I've never broken it that bad as a user before | 14:00 |
| fungi | and it's running with memory mostly full trying to maintain cache memory and buffers, while not able to page out anything | 14:00 |
| *** ramishra has quit IRC | 14:00 | |
| fungi | so more memory _and_ a swapfile might help too? | 14:01 |
| pabelanger | fungi: Ah, I missed no swap. so, 4C/8GB ? | 14:01 |
| *** e0ne has quit IRC | 14:01 | |
| *** e0ne has joined #openstack-infra | 14:02 | |
| fungi | pabelanger: yeah, if we can go with 8gb ram (and hopefully more cores) on the replacement, and add a swapfile, performance would likely improve some | 14:02 |
| *** gouthamr has joined #openstack-infra | 14:02 | |
| pabelanger | fungi: k, let me fresh coffee and look into that | 14:02 |
| *** chlong has joined #openstack-infra | 14:04 | |
| fungi | wonder whether we should do the same in the other 3 citycloud regions for consistency? weird it should be that loaded though... we're only booting up to 50 nodes there | 14:04 |
| fungi | also we have both lon1 and sto2 out of commission at the moment | 14:05 |
| fungi | i didn't realize we were down two citycloud regions, thought it was just sto2 | 14:06 |
| pabelanger | I think we can try sto2 again, and see how it operates. lon1 needs to get looked at, terrible IO there | 14:06 |
| *** ramishra has joined #openstack-infra | 14:07 | |
| fungi | okay, let's see about getting sto2 back into rotation there | 14:07 |
| fungi | especially since it looks like we could be without ovh for another week still | 14:07 |
| *** esberglu has joined #openstack-infra | 14:08 | |
| pabelanger | ya | 14:09 |
| fungi | also, looks like the added periodic jobs for stable/ocata are hitting us pretty hard on limited capacity (37 still queued from 8 hours ago) | 14:09 |
| pabelanger | fungi: I think mnaser also wanted to try vexxhost too | 14:10 |
| *** caphrim007 has quit IRC | 14:10 | |
| *** annegentle has joined #openstack-infra | 14:10 | |
| mnaser | did we land the glean changes with updates images? | 14:11 |
| pabelanger | ya, should be live now | 14:11 |
| pabelanger | I can boot one quickly to confirm | 14:11 |
| *** isaacb has joined #openstack-infra | 14:11 | |
| fungi | activity was already ramping up a few hours prior, but zuul was keeping up until the periodic jobs hit we've been backlogged ever since | 14:11 |
| mnaser | ok let me submit a small bump and we'll see how it goes | 14:11 |
| *** sree has quit IRC | 14:12 | |
| *** sree has joined #openstack-infra | 14:12 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Skip additional tests for Cinder doc changes https://review.openstack.org/492630 | 14:13 |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack-infra/project-config master: [magnum] Make swarm-mode and k8s tests -nv https://review.openstack.org/493556 | 14:13 |
| *** tushar has quit IRC | 14:13 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Set vexxhost max-servers to 10 and change instance type https://review.openstack.org/493557 | 14:14 |
| *** tushar has joined #openstack-infra | 14:14 | |
| mnaser | fungi pabelanger ^ | 14:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Temporarily stop using citycloud-sto2" https://review.openstack.org/493558 | 14:14 |
| tushar | hi All , Facing issue while accessing the nodepool nodes in thirdparty CI setup | 14:15 |
| tushar | Is there any change in DIB image ? for which we need to changes our setup accordingly? | 14:16 |
| pabelanger | mnaser: is the flavor correct? | 14:16 |
| mnaser | yes this is 8 core / 8 gb memory which should perform better pabelanger | 14:16 |
| tushar | With the the old DIB image(3-4 days old) I can aceess the nodes , but issue is with latest DIB image | 14:17 |
| mnaser | looks like two dedicated cores arent well saturated | 14:17 |
| fungi | tushar: there was a new glean release late last week which changed how labels for vfat configdrives are checked, but i don't think that should have had any impact (pretty sure vfat configdrives were simply not working at all previously) | 14:17 |
| *** sree has quit IRC | 14:17 | |
| pabelanger | mnaser: Hmm, I am unable to boot v1-performance-2-80gb | 14:17 |
| pabelanger | not enough hosts avaiable | 14:18 |
| mnaser | pabelanger you'll want to try/go for v1-standard-8 | 14:18 |
| pabelanger | mnaser: ah | 14:18 |
| fungi | tushar: unless maybe something about the configdrive data in your cloud is wrong and glean wasn't able to load that broken data until vfat support was corrected? | 14:18 |
| mnaser | taking that out of rotation because our experimenting showed that dedicated cores apparently don't end up being saturated | 14:18 |
| *** martinkopec has joined #openstack-infra | 14:18 | |
| *** xyang1 has joined #openstack-infra | 14:19 | |
| tushar | Fungi :Thanks.. can you please provide any refernce for this change...I can look into | 14:19 |
| pabelanger | k, booting now | 14:19 |
| pabelanger | cool | 14:20 |
| pabelanger | glean worked | 14:20 |
| pabelanger | was able to SSH in as root | 14:20 |
| fungi | tushar: has anybody else in #openstack-third-party-ci seen similar issues? tushar things which changed in glean are the commits you see between the 1.9.1 and 1.9.2 tags at http://git.openstack.org/cgit/openstack-infra/glean/log/ | 14:20 |
| tushar | fungi : I posted query in #openstack-third-party-ci , no one replied for the similar issue | 14:22 |
| *** marst has joined #openstack-infra | 14:23 | |
| *** funzo has joined #openstack-infra | 14:24 | |
| bogdando | pabelanger: hi. If I correctly understand the classifier page http://status.openstack.org/elastic-recheck/data/others.html , we may create generic bugs to bind failures to known events, even if there is a technical solution for the root cause, like with cached vs direct connections. Am I missing something? | 14:27 |
| *** rbrndt has joined #openstack-infra | 14:27 | |
| *** tnovacik has joined #openstack-infra | 14:27 | |
| *** funzo has quit IRC | 14:29 | |
| bogdando | or shall I use no graph option? | 14:30 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Zun] Move etcd dsvm job back to check queue https://review.openstack.org/493564 | 14:35 |
| *** sree has joined #openstack-infra | 14:37 | |
| pabelanger | bogdando: Ya, I think there is some overlap with existing queries we have setup today. 493520 for example, should just be renamed to something like invalid repo file, since baseurl is empty. And if we need to modify existing queries, we can | 14:41 |
| pabelanger | bogdando: clouds.centos.org should also be added to our reverse proxy cache, that one we can likely fix without landing E-R. And your slow networking one, it likely useful to help identify mirror issues | 14:42 |
| bogdando | pabelanger: yeah. I hope to reduce the | 14:44 |
| bogdando | recheck numbers* | 14:44 |
| pabelanger | bogdando: ya, mwhahaha and I made some good progress last week on 3 of them. your empty baseurl DLRN was new to me | 14:45 |
| *** LindaWang has quit IRC | 14:50 | |
| AJaeger | fungi, you mentioned the backlog with periodic jobs - can we merge https://review.openstack.org/493080 for periodic translations for pike? We could ask ianychoi to close newton translations now | 14:55 |
| *** annegentle has quit IRC | 14:57 | |
| *** yamahata has joined #openstack-infra | 14:59 | |
| *** ramishra has quit IRC | 15:00 | |
| *** rcernin has quit IRC | 15:00 | |
| *** gcb has quit IRC | 15:00 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/elastic-recheck master: Track issues with invalid repo files, since baseurl is empty https://review.openstack.org/493520 | 15:01 |
| pabelanger | fungi: Ya, our make_swap.sh file needs to be updated. http://paste.openstack.org/show/618321/ we are no longer setting up swap in some cases | 15:02 |
| pabelanger | fungi: that is from booting replacement mirror.kna1.citycloud.o.o | 15:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: WIP use new keystoneauth version discovery https://review.openstack.org/493582 | 15:06 |
| bogdando | pabelanger: thanks! updated the pr | 15:07 |
| bogdando | folks, PTAL https://review.openstack.org/#/c/492178/ this helps to classify e-r patterns , at least for tripleo-ci | 15:08 |
| *** baoli has quit IRC | 15:09 | |
| *** isaacb has quit IRC | 15:10 | |
| *** baoli has joined #openstack-infra | 15:12 | |
| *** amotoki is now known as amotoki__away | 15:14 | |
| *** ccamacho has left #openstack-infra | 15:15 | |
| openstackgerrit | Merged openstack-infra/project-config master: Setup translations for Pike https://review.openstack.org/493080 | 15:20 |
| *** shardy has quit IRC | 15:22 | |
| *** funzo has joined #openstack-infra | 15:22 | |
| *** shardy has joined #openstack-infra | 15:23 | |
| pabelanger | fungi: have replacement mirror for kna1 online, about to change DNS entries | 15:25 |
| fungi | pabelanger: awesome, i'll try to keep an eye on things | 15:26 |
| *** sree has quit IRC | 15:27 | |
| *** sree has joined #openstack-infra | 15:27 | |
| pabelanger | #status log mirror.kna1.citycloud.openstack.org DNS entry updated to 91.123.202.15 | 15:29 |
| openstackstatus | pabelanger: finished logging | 15:29 |
| *** caphrim007 has joined #openstack-infra | 15:31 | |
| *** sree has quit IRC | 15:32 | |
| fungi | pabelanger: after deleting the old mirror, remember to delete the cached ansible inventory on the puppetmaster and clear/reaccept the ssh host key as root | 15:33 |
| pabelanger | fungi: yup, thanks | 15:33 |
| jrich | kia sorento vs barbecue grill + cig: http://myfox8.com/2017/08/13/couple-transporting-grill-injured-after-suv-explodes-when-wife-lights-up-cigarette/?utm_source=fark&utm_medium=website&utm_content=link&ICID=ref_fark | 15:35 |
| *** dprince has quit IRC | 15:35 | |
| fungi | jrich: i'm guessing that was meant for a different channel | 15:36 |
| *** felipemonteiro has joined #openstack-infra | 15:36 | |
| clarkb | pabelanger: fungi we bumping flavor for network in kna1? | 15:37 |
| fungi | clarkb: for more ram and cpu i think, judging from the cacti graphs | 15:37 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: WIP use new keystoneauth version discovery https://review.openstack.org/493582 | 15:38 |
| fungi | clarkb: though odds are it's the rootfs being really slow there, so not sure it will help | 15:38 |
| pabelanger | clarkb: fungi: ya, I am keeping original around in case we want to roll back | 15:39 |
| jrich | it was, appologies. | 15:39 |
| fungi | and turned rax ord back up an hour or so ago, keeping an eye out for the network issues we saw with the old mirror in case that didn't fix things | 15:39 |
| *** david-lyle has joined #openstack-infra | 15:41 | |
| clarkb | cool | 15:41 |
| fungi | i did some analysis of htcacheclean performance there over the weekend | 15:41 |
| *** felipemonteiro has quit IRC | 15:41 | |
| fungi | manually expiring cache contents older than 15 days got us under 50% and then running htcacheclean with -r and no -n took it down to around 45% utilization. after that, rerunning the exact htcacheclean command from the cronjob under time (with apache running again but no actual load on the mirror) took 4 hours 3 minutes to complete | 15:42 |
| fungi | no idea how long it will take with the mirror in active heavy use again | 15:43 |
| clarkb | and thats with old dir structure right? | 15:43 |
| fungi | correct | 15:43 |
| clarkb | hopefully new one helps too | 15:43 |
| fungi | no idea whether it will or not. i saw some discussions in various forums and mailing lists talking about optimizing htcacheclean performance by switching to a flatter directory structure instead of a deeper one | 15:44 |
| *** chlong_ has joined #openstack-infra | 15:45 | |
| *** chlong has quit IRC | 15:45 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create site_static secret https://review.openstack.org/492671 | 15:45 |
| fungi | though the bandiwdth cap seems to no longer be in effect at least: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=3063&rra_id=all | 15:45 |
| jeblair | fungi, clarkb: wasn't parte of the idea that you could parallelize htcacheclean with the new dir structure? | 15:45 |
| jeblair | fungi: oh wow, yeah, i guess bandwidth was indeed a problem as suspected :) | 15:46 |
| fungi | jeblair: oh, i hadn't thought of that, but i guess we could fairly easily now that you mention it | 15:46 |
| fungi | and leverage more cpus (maybe?) | 15:46 |
| *** links has joined #openstack-infra | 15:47 | |
| *** e0ne has quit IRC | 15:47 | |
| pabelanger | jeblair: fungi: Ya, I found a bash script on the interwebs to do just that. Requires 4096 top-level dirs currently, but can be modified | 15:47 |
| openstackgerrit | Merged openstack-infra/project-config master: Set vexxhost max-servers to 10 and change instance type https://review.openstack.org/493557 | 15:47 |
| *** Guest31152 has quit IRC | 15:47 | |
| fungi | also possible that the rebuild landed us on a host with less i/o contention: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=3020&rra_id=all | 15:48 |
| openstackgerrit | Jamey McCabe proposed openstack-infra/irc-meetings master: Adding new meeting for LCOO WG https://review.openstack.org/492220 | 15:48 |
| pabelanger | on the 10th is when we added locking to htcacheclean, to force 1 process | 15:48 |
| fungi | right, but utilization over the next day or so was still pretty high compared to what it is now | 15:49 |
| pabelanger | ya | 15:49 |
| fungi | the gap in the graph late on the 11th looks like probably when i replaced the mirror | 15:50 |
| *** vhosakot has joined #openstack-infra | 15:50 | |
| pabelanger | jeblair: mind looking at 492671 again. Thinking we might want to store more information into our secret, but some of it isn't 'encrypted'. So, unsure if we want to do something like that for sites or just use normal variables | 15:50 |
| fungi | since system/user go away at that point and we just have iowait (from me doing find and htcacheclean without -n) | 15:50 |
| jeblair | pabelanger: yeah, i threw that suggestion out there without having any idea which we would prefer. :) | 15:51 |
| jeblair | pabelanger: we probably won't know until we try | 15:51 |
| jeblair | (each option has pros and cons) | 15:51 |
| jeblair | i like that this has a nice tidy package for all the information | 15:51 |
| pabelanger | jeblair: ya, I see comment 2 now, I missed that | 15:51 |
| pabelanger | ya, same | 15:51 |
| clarkb | jeblair: ya smarter dir cleaning when we have 4k instead of 200k dir to list | 15:52 |
| jeblair | the downside is if we want to share that key, it's a little more awkward. however, maybe even if the key is the same, we *should* duplicate the secret just to make it easier to maintain. | 15:52 |
| *** Apoorva has joined #openstack-infra | 15:52 | |
| pabelanger | Ya, was thinking that too | 15:52 |
| jeblair | pabelanger: anyway, my vote is that we make sure other infra-root folks have at least seen https://review.openstack.org/492671 so they know what's up, and then try it and see what we like/dislike. | 15:53 |
| pabelanger | wfm | 15:54 |
| openstackgerrit | Jamey McCabe proposed openstack-infra/irc-meetings master: Adding new meeting for LCOO WG https://review.openstack.org/492220 | 15:55 |
| *** dprince has joined #openstack-infra | 15:56 | |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack-infra/project-config master: [magnum] Make swarm-mode and k8s tests -nv https://review.openstack.org/493556 | 15:56 |
| openstackgerrit | Jamey McCabe proposed openstack-infra/irc-meetings master: Adding new meeting for LCOO WG https://review.openstack.org/492220 | 15:57 |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack-infra/project-config master: [magnum] Make swarm-mode and k8s tests -nv https://review.openstack.org/493556 | 15:58 |
| *** tesseract has quit IRC | 16:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean master: Simplify glean.sh options https://review.openstack.org/493329 | 16:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean master: Remove config_exists function from glean.sh https://review.openstack.org/493330 | 16:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean master: Add --skip-dns option https://review.openstack.org/493327 | 16:03 |
| *** psachin has joined #openstack-infra | 16:03 | |
| pabelanger | clarkb: fungi: mordred: do you mind adding ^ to your review pipeline. I'd like to start testing systemd config files to start passing --skip-dns option to glean | 16:04 |
| fungi | jeblair: seems fine in all, though i get the impression we're still feeling out what sorts of non-secret data can simplify the playbooks by including them in a secret? | 16:04 |
| *** Hal has joined #openstack-infra | 16:04 | |
| *** Hal is now known as Guest80741 | 16:04 | |
| clarkb | I havent had a chance to read it yet but isnt that what normal non secret vars are for? | 16:05 |
| *** jpena is now known as jpena|off | 16:05 | |
| *** Swami has joined #openstack-infra | 16:05 | |
| fungi | secrets have the ability to contain encrypted and non-encrypted variables for convenience | 16:06 |
| fungi | we'd already discussed examples where the username was kept in the secret but using a plaintext value | 16:06 |
| pabelanger | ya, right now the patch would allow {{ site_static.private_ssh_key }} and {{ site_static.ssh_user }}. Only the ssh key is encrypted | 16:07 |
| fungi | so it'll be interesting to see what sorts of non-secret data we standardize on agglomerating into secrets | 16:07 |
| pabelanger | so grouping of public / private into same dict | 16:07 |
| clarkb | I see data related to the secret that isnt secret as well | 16:07 |
| pabelanger | at least, that is how I currently understand it | 16:07 |
| *** jpena|off is now known as jpena | 16:08 | |
| fungi | right, "related" being a bit of a relative/fuzzy distinction unfortunately | 16:08 |
| *** egonzalez has quit IRC | 16:08 | |
| jeblair | yep. and ssh keys are even fuzzier than user/password. | 16:08 |
| pabelanger | clarkb: not related, spacex launch in 25mins | 16:08 |
| jeblair | like, sticking the pypi username and password onto a single 'secret' is really obvious. how much of the host key, username, etc, should be attached to the ssh key for static.o.o is less obvious. :) | 16:09 |
| *** camunoz has quit IRC | 16:09 | |
| fungi | today's spacex launch is out of kennedy. i'm bummed the next launch out of the mid-atlantic spaceport (next door to me on wallop island) is taking place while i'm out of town for the ptg | 16:12 |
| fungi | er, wallops island | 16:12 |
| pabelanger | just seen a tweet that pleia2 is currently there! | 16:12 |
| fungi | right on! | 16:12 |
| clarkb | pabelanger: ooh thanks I wouldve missed it | 16:12 |
| *** Julien-zte has quit IRC | 16:12 | |
| *** tushar_ has joined #openstack-infra | 16:13 | |
| *** Julien-zte has joined #openstack-infra | 16:13 | |
| rybridges | Hey guys. Do you think we could please merge this today? https://review.openstack.org/#/c/492693/ | 16:19 |
| *** yamamoto has quit IRC | 16:21 | |
| fungi | pabelanger: on the glean --skip-dns option, would we plumb that through nodepool configuration? i'm curious how we'd set it on the daemon startup configuration on a per-provider basis since we don't use different images | 16:22 |
| fungi | pabelanger: oh, i misread the commit message. when you say openstack-infra you mean all our providers, not infra-cloud | 16:22 |
| fungi | er, rather, not _just_ infra-cloud | 16:23 |
| pabelanger | fungi: I was considering doing it provider wide for centos-7 for example. But, I actually think we could just apply it to all images in our base nodepool element, since everything should be using unbound service | 16:23 |
| *** aarefiev is now known as aarefiev_afk | 16:25 | |
| fungi | jeblair: attempting to load 492693 crashes for me on latest gertty release. are you able to reproduce locally or want traceback details? | 16:25 |
| clarkb | I've got the twins on the couch watching youtube stream of launch now :) | 16:26 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: Consolidate client version checks in an utility method https://review.openstack.org/493598 | 16:26 |
| jeblair | fungi: fails for me too | 16:26 |
| clarkb | pabelanger: but we don't have systemd on all images (yet) | 16:26 |
| fungi | okay. something about markdown parsing looks like. wonder if it's trying to interpret the commit message as markdown? | 16:27 |
| clarkb | pabelanger: so at least the trusty images would be odd ones outright? | 16:27 |
| fungi | er, markUP (not markDOWN) | 16:27 |
| pabelanger | clarkb: Ya, trusty is odd one out, but currently debuntu doesn't have this DNS issue. only centos / fedora | 16:27 |
| pabelanger | aka SYSTEMD! | 16:27 |
| pabelanger | :) | 16:27 |
| jeblair | fungi: looks like it's related to the reviername, so i'm gussing it's related to 'pheartheceal@gmail.com' as a gerrit username | 16:28 |
| fungi | jeblair: oh, indeed | 16:29 |
| jeblair | fungi: oh, i see, there is no 'name' for that account, so gerrit webui is falling back on email | 16:30 |
| fungi | wacky. i didn't realize the accounts.full_name field was allowed to be blank | 16:31 |
| jeblair | fungi: i didn't think it was either | 16:31 |
| fungi | jeblair: i see it also does a slashdot and substitutes the name "anonymous coward" in the comments header | 16:33 |
| jeblair | :) | 16:34 |
| jeblair | fungi: interestingly, gertty has code to handle this is the case of authors and review comments, but not review votes | 16:35 |
| *** electrofelix has quit IRC | 16:36 | |
| *** ykarel has quit IRC | 16:37 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: Consolidate client version checks in an utility method https://review.openstack.org/493598 | 16:38 |
| openstackgerrit | James E. Blair proposed openstack/gertty master: Handle approvals with no name https://review.openstack.org/493601 | 16:39 |
| jeblair | fungi: ^ | 16:39 |
| fungi | jeblair: confirmed, there's one account with that e-mail address as its preferred_email and has a full_name of NULL in the accounts table | 16:40 |
| clarkb | I wonder if that is inhreited from ubuntu openid? | 16:41 |
| openstackgerrit | Merged openstack/gertty master: Change usage of exit() to sys.exit() https://review.openstack.org/492622 | 16:41 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Send logs size to grahite site https://review.openstack.org/481159 | 16:42 |
| jeblair | today is one of those mondays where i close a terminal window which has been pinging google at least all weekend. | 16:43 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: WIP use new keystoneauth version discovery https://review.openstack.org/493582 | 16:45 |
| fungi | jeblair: interestingly, your "anonymous coward" path seems to not be exercised, as the comments show up with a full name of "None" now | 16:45 |
| jeblair | fungi: so they do | 16:46 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: WIP use new keystoneauth version discovery https://review.openstack.org/493582 | 16:47 |
| *** spzala has quit IRC | 16:49 | |
| pabelanger | clarkb: did you catch the discussion I had this morning with fungi about make_swap.sh not working on mirrors | 16:49 |
| clarkb | pabelanger: briefly, its becaues we don't have a second disk device in all clouds | 16:50 |
| *** jpena is now known as jpena|off | 16:50 | |
| pabelanger | clarkb: Ya, thoughts on what we could do? Just dump the swap on the primary? | 16:50 |
| clarkb | we'd probably want to do like devstack-gate does and use a swap file? | 16:51 |
| clarkb | since repartitioning root device live is probably not something we want to do on every launch | 16:51 |
| fungi | right, i was suggesting going with a swapfile in that situation | 16:52 |
| pabelanger | okay, I agree swapfile is likely best approach | 16:52 |
| pabelanger | I'll see about updating make_swap.sh | 16:52 |
| fungi | i think we're getting some suboptimal performance around cannibalizing buffers memory when active memory spikes | 16:53 |
| fungi | and having somewhere for it to page things even a tiny amount could help smooth that out | 16:53 |
| *** tnovacik has quit IRC | 16:53 | |
| *** yamahata has quit IRC | 16:54 | |
| clarkb | should be able to reuse what is in devstack-gate | 16:55 |
| fungi | agreed | 16:55 |
| clarkb | it tries to be smart about how it allocates the space particularly on ext4 which is nice | 16:55 |
| fungi | wonder if there's a sane way for us to use that in both places without carrying duplicate copies of the same in different repos | 16:56 |
| jeblair | fungi: ansible role? | 16:57 |
| pabelanger | Ya, ansible role makes most sense | 16:57 |
| *** derekh has quit IRC | 16:57 | |
| fungi | seems like that's what we eventually want for devstack-gate anyway, so... YES! | 16:58 |
| *** rcernin has joined #openstack-infra | 16:58 | |
| clarkb | ya I don't think its an ansible role yet becaues it is called out of setup_workspace not setup_host but that can/should be moved then updated to be role | 16:58 |
| fungi | clarkb: jeblair: reverse openid lookup says https://launchpad.net/~jacobe is the lp profile corresponding to the gerrit account with the NULL accounts.full_name | 17:01 |
| *** jamesden_ has quit IRC | 17:01 | |
| clarkb | bogdando: 492178 is not an e-r change, it is a tripleo-ci change | 17:01 |
| bogdando | clarkb: right, sorry for noise | 17:02 |
| clarkb | fungi: that use appears to have a public full name in lp at least | 17:03 |
| fungi | clarkb: jeblair: _though_ the gerrit openid requests full name and email address but ubuntu one lets you uncheck them, so maybe that's what happened | 17:03 |
| clarkb | ah | 17:03 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Add jobvar and rolevar directives https://review.openstack.org/493248 | 17:03 |
| fungi | i haven't tried creating a new test user to see if that happens, just looking at the personal data request page when i log out of gerrit and start to log back in again | 17:04 |
| fungi | so seems a plausible explanation | 17:04 |
| fungi | now if i just new the mysql query magic for returning rows where some column =NULL | 17:04 |
| fungi | s/new/knew/ | 17:04 |
| *** links has quit IRC | 17:05 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Skip additional tests for Cinder doc changes https://review.openstack.org/492630 | 17:06 |
| *** lucasagomes is now known as lucas-afk | 17:06 | |
| *** sshnaidm has quit IRC | 17:06 | |
| *** jpich has quit IRC | 17:06 | |
| pabelanger | clarkb: want to review, +3 493558 ? Want to keep an eye on it today and see how it preforms | 17:07 |
| AJaeger | pabelanger, clarkb , I was just going whether I should +2 or +3 - I can +3... ^ | 17:10 |
| clarkb | pabelanger: done | 17:10 |
| clarkb | AJaeger: I may have just beat you to it :) | 17:10 |
| AJaeger | ok, then I abstain ;) | 17:10 |
| AJaeger | clarkb: no wories | 17:10 |
| jeblair | i tagged zuul-sphinx 0.1.5 | 17:11 |
| clarkb | pabelanger: looks like nodepool is going full steam ahead so we should have info on reliability of sto2 quickly | 17:13 |
| pabelanger | clarkb: Agree | 17:15 |
| AJaeger | we have 3 hold nodes - are those needed? | 17:15 |
| fungi | at least the waiting jobs backlog seems to be trending back downward over the last couple hours | 17:15 |
| pabelanger | would be curious if DNS issues were limited to centos | 17:15 |
| pabelanger | I cannot remember | 17:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Disable py2 dsvm on nodepool feature/zuulv3 branch https://review.openstack.org/492629 | 17:15 |
| jeblair | AJaeger: probably mine, i'll check and delete | 17:15 |
| jeblair | AJaeger: 2 for me (testing infracloud bandwidth issues). 1 for clarkb testing pypi caching proxy. | 17:16 |
| clarkb | mine can be deleted | 17:16 |
| AJaeger | thanks, jeblair | 17:16 |
| jeblair | clarkb: done | 17:16 |
| clarkb | thanks | 17:17 |
| fungi | mgagne: any chance you're around? did you see the case of the phantom instance i tracked down saturday? summary is that looks like 191beb1c-a527-4deb-a8f9-37b853b981cc may have existed for a some days after nova thought it had been deleted, and kept fighting other instances for ip address 198.72.124.233 | 17:19 |
| fungi | mgagne: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-08-12.log.html#t2017-08-12T13:32:58 | 17:19 |
| fungi | i can't ping that address now, so suspect it may finally be gone | 17:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Temporarily stop using citycloud-sto2" https://review.openstack.org/493558 | 17:20 |
| fungi | well, something just now started responding to ping there again | 17:21 |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/elastic-recheck master: Track issues with invalid repo files, since baseurl is empty https://review.openstack.org/493520 | 17:21 |
| fungi | but it's rejecting my ssh key so i expect it's not the ghost | 17:21 |
| fungi | last job result we saw logged to its console logfile was 05:37 utc saturday | 17:22 |
| *** yamamoto has joined #openstack-infra | 17:22 | |
| fungi | infra-root: given we saw that behavior in infra-cloud too... i wonder whether we should attempt to explicitly halt single-use nodes after jobs complete, as a safeguard against similar accidental reuse? | 17:23 |
| clarkb | that may not be a bad idea. Possibly related I've noticed that using -minimal images locally virsh cannot gracefully stop them they have to be destroyed (and experienced this cleaning up nodes in infracloud too). Posslbe that affects clouds' ability to shut them down? | 17:25 |
| *** sambetts is now known as sambetts|afk | 17:25 | |
| clarkb | guessing it is a lack of acpi packages to handle the acpi events for graceful shutdown | 17:25 |
| clarkb | acpi, acpid, and acpi-support look like the packages we want on ubuntu | 17:26 |
| *** bogdando has quit IRC | 17:26 | |
| pabelanger | fungi: how are you propsing for zuulv2.5? In zuulv3.o.o we could update our post playbook in base job to handle that after finally log collect task | 17:26 |
| jeblair | pabelanger, fungi: note in zuulv3, the ssh key swap we do will prevent reuse | 17:27 |
| fungi | pabelanger: more just a consideration for the future | 17:27 |
| fungi | jeblair: oh, right, so good enough then | 17:27 |
| jeblair | we can *also* halt; it would be easy to do | 17:27 |
| *** spzala has joined #openstack-infra | 17:27 | |
| fungi | doesn't seem especially necessary | 17:27 |
| fungi | since we won't ignore host keys and will preauthorize unique access keys | 17:28 |
| pabelanger | ya, if we think we are leaking, halting would help limited raised resources | 17:28 |
| pabelanger | Totally forgot about one time keys in zuulv3 | 17:28 |
| openstackgerrit | Ryan proposed openstack-infra/bindep master: Add ability to list all deps https://review.openstack.org/492693 | 17:28 |
| jeblair | for now, i think i'd rather just play the odds for the next month, since i don't think it would be entirely straightforward to do it with zuul v2.5 | 17:28 |
| fungi | jeblair: right, it was more of a post-v3 thought anyway | 17:28 |
| jeblair | (we *can* do it; i just expect it to take ~1 day of work) | 17:28 |
| AJaeger | pabelanger: could you review https://review.openstack.org/493003 again, please? That's "Remove OpenStack/Ceph/Virt repo from CentOS". | 17:28 |
| fungi | it seems to crop up infrequently | 17:28 |
| rybridges | pabelanger: Unit test added here https://review.openstack.org/#/c/492693/ | 17:29 |
| *** yamamoto has quit IRC | 17:29 | |
| *** annegentle has joined #openstack-infra | 17:29 | |
| *** jtomasek has quit IRC | 17:31 | |
| openstackgerrit | Ryan proposed openstack-infra/bindep master: Add ability to list all deps https://review.openstack.org/492693 | 17:31 |
| *** sree has joined #openstack-infra | 17:32 | |
| clarkb | AJaeger: pabelanger we just need to update devstack-gate to install the repo and ovs properly before that change and double check that devstack itself will enable the repo if it needs it | 17:32 |
| *** jtomasek has joined #openstack-infra | 17:32 | |
| pabelanger | AJaeger: ya, we cannot land that until A) we remove openvswitch from devstack, clarkb is looking at. B) have devstack-gate add back ocata RPM package, disable repo, then yum install openvswitch only from said repo. But I still think we are too close to R+0 to implement this. | 17:33 |
| clarkb | I'm pretty sure removing ovs from devstack-gate isn't going to happen any time soon | 17:33 |
| clarkb | we can keep working at that as we go but the simpler fix for now is to install the repothere if we need it | 17:33 |
| AJaeger | clarkb: do you want to WIP or -2 the change for now? | 17:34 |
| pabelanger | Ya, I think B is likley the best approach, but not sure I can find time between now and R+0. And since devstack-gate is getting updated for zuulv3, seems like we could make the change then | 17:34 |
| clarkb | it turns out a lot of stuff assumes ovs for that overlay | 17:34 |
| clarkb | and changing that will be slow process if we do it | 17:34 |
| pabelanger | so, PTG is when I can likely hack on it | 17:34 |
| clarkb | pabelanger: I dunno its self testing and not like we are changing the repo | 17:34 |
| *** tosky has quit IRC | 17:34 | |
| clarkb | it should be a completely safe change if tests pass | 17:35 |
| clarkb | we might want to not change the images until then though | 17:35 |
| clarkb | but d-g we should be able to change if tests pass | 17:35 |
| pabelanger | agree, d-g change could land now | 17:35 |
| mgagne | fungi: I'm here now. Is your issue solved? | 17:36 |
| pabelanger | I just haven't looked at updating install_package to support yum install foo --enable-repo=bar logic | 17:36 |
| *** sree has quit IRC | 17:36 | |
| mnaser | pabelanger any way to check if jobs are healthily running? | 17:37 |
| fungi | mgagne: we think so, but hard to know since the api obviously can't tell us | 17:37 |
| *** yamahata has joined #openstack-infra | 17:37 | |
| *** jamesdenton has joined #openstack-infra | 17:37 | |
| mgagne | fungi: ok, I can take a look | 17:37 |
| pabelanger | mnaser: we can check logstash.o.o again | 17:37 |
| *** ykarel has joined #openstack-infra | 17:38 | |
| mnaser | pabelanger looks like i see 490949 only and it fialed | 17:38 |
| *** martinkopec has quit IRC | 17:38 | |
| mnaser | non voting job though | 17:39 |
| fungi | mgagne: when we saw it happen in infra-cloud we ended up comparing lists of instance uuids per compute node from the nova api to what virsh saw locally on each of them to locate "leaked" virtual machine instances. no idea if you already have some sort of cleanup automation running in your operation given the substantially larger scale | 17:39 |
| pabelanger | mnaser: ya, lets find another | 17:39 |
| mgagne | fungi: if you have the MAC address, that would be helpful. I see that the IP responds to ping but isn't present in neutron port-list | 17:40 |
| mnaser | i think we might be able to get that mgagne | 17:40 |
| fungi | mgagne: i bet we have that... checking the old log to see if we printed it | 17:40 |
| mnaser | let me try to find the messy job | 17:40 |
| mgagne | I could spin a VM within your network but I don't think it's a good idea tbh =) | 17:40 |
| mgagne | fungi: can you SSH in a slave and check? to get the latest MAC | 17:41 |
| mgagne | otherwise I can find a way with our netadmin team | 17:41 |
| clarkb | I will try a dib build of ubuntu-minimal with those acpi packages installed to see if that fixes virsh things | 17:41 |
| mnaser | http://logs.openstack.org/12/491312/4/gate/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/909ae4a/console.html | 17:41 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: WIP: send graphite stats from ARA https://review.openstack.org/480121 | 17:41 |
| mnaser | mgagne you could look for vm name ubuntu-xenial-internap-mtl01-10338966 in the db records, check the host it was at and then verify that hypervisor | 17:42 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: Add support for updating initramfs images in CI https://review.openstack.org/479900 | 17:42 |
| fungi | mnaser: well, i dug up the uuid so that's likely at least as easy to find in logs as the instance name | 17:43 |
| mnaser | fungi oh yes thats true as well | 17:44 |
| mnaser | probably more accurate as well | 17:44 |
| fungi | the original job which ran on it doesn't seem to include our network diag builder so didn't echo addressing in the start of the console log unfortunately | 17:44 |
| fungi | but i'll try to get arp for that v4 address from another node in the same network and see if i get multiple responses | 17:45 |
| *** rybridges has quit IRC | 17:45 | |
| AJaeger | clarkb, pabelanger : Do you want to block https://review.openstack.org/#/c/493003/ for now? | 17:45 |
| pabelanger | I can -1 with comments | 17:46 |
| clarkb | ya lets wip it. | 17:46 |
| AJaeger | thanks | 17:46 |
| *** xarses_ has joined #openstack-infra | 17:47 | |
| fungi | mgagne: current arp reply i see for that address is fa:16:3e:f3:de:1e | 17:49 |
| fungi | no indication in dmesg that i got conflicting responses either | 17:50 |
| fungi | and since there's a server there refusing my ssh key presently, i'm going to assume it's not ours | 17:51 |
| *** xarses has quit IRC | 17:51 | |
| pabelanger | clarkb: fungi: I'd like to experiment more with bundler install and reverse proxy cache, do you mind reviewing and +2 / +3 493316 | 17:51 |
| fungi | mgagne: and that the phantom instance was most likely cleaned up at some point over the past couple of days (possibly even by the time i was looking into it on saturday) | 17:51 |
| mgagne | fungi: the IP responds to ping and isn't registered in Neutron. I'm tracking it down | 17:52 |
| mgagne | fungi: f51b07e9-6781-47e6-8a23-81669655b30e | 17:53 |
| mgagne | destroyed the instance | 17:54 |
| mgagne | and now IP stopped responding to ping | 17:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Temporarily start using the public registry again" https://review.openstack.org/493630 | 17:54 |
| *** shardy has quit IRC | 17:54 | |
| *** jamesdenton has quit IRC | 17:54 | |
| *** jamesdenton has joined #openstack-infra | 17:55 | |
| fungi | mgagne: interesting, that also doesn't correspond to the uuid we saw for (what i think was) the original instance | 17:55 |
| fungi | i'll go hunting in our logs to see if that was ever one of ours | 17:56 |
| mgagne | who else could it belong to? =) | 17:56 |
| fungi | mgagne: or at least that was not the uuid nova reported to us when the instance was created | 17:57 |
| mgagne | ubuntu-xenial-2-node-internap-mtl01-8471395-533080 | 17:57 |
| fungi | but also the fact that the root account on it didn't allow my ssh key (which is standard for our instances) is sort of odd | 17:58 |
| *** xarses_ has quit IRC | 17:58 | |
| fungi | oh! maybe we wouldn't see it on a subnode? | 17:58 |
| mgagne | created_at: 2017-04-18 18:31:29 | 17:58 |
| mgagne | so it's very old stuff? | 17:59 |
| fungi | wow, no wonder that uuid didn't show up in our debug logs | 17:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Set template-hostname for vexxhost https://review.openstack.org/493633 | 17:59 |
| *** markvoelker has quit IRC | 17:59 | |
| *** markvoelker has joined #openstack-infra | 18:00 | |
| fungi | and yes, that predates when we moved to authorizing our accounts for root login rather than installing individual accounts, so explains the key being rejected (i didn't even think to try my account since i wouldn't have expected to have one on nodes for months now) | 18:00 |
| mgagne | We usually have a script checking consistency between compute nodes and database. But I think due to the very high rate of creation/deletion, it had too many false positives so we probably disabled it. | 18:00 |
| *** EricGonczer_ has quit IRC | 18:01 | |
| *** dtantsur is now known as dtantsur|afk | 18:01 | |
| mgagne | maybe we could run it manually once or twice to find the remaining phantoms | 18:01 |
| *** ykarel has quit IRC | 18:02 | |
| *** tosky has joined #openstack-infra | 18:03 | |
| jeblair | fungi: hey! for some reason my firefox has been returning OCSP errors for review and storyboard for the past hour | 18:04 |
| jeblair | i'm trying to convince it to use more words telling me what's wrong, but am not faring well | 18:04 |
| *** markvoelker has quit IRC | 18:04 | |
| fungi | jeblair: you might try force reloading those pages in case your browser cached the old certs which were due to expire around now | 18:04 |
| jeblair | oh, huh that's a thing? | 18:05 |
| fungi | given i replaced instead of renewing them | 18:05 |
| jeblair | wow. | 18:05 |
| jeblair | apparently that worked | 18:05 |
| jeblair | i don't understand the mechanics of this. | 18:05 |
| jeblair | i'm going to see if my laptop shows similar behavior | 18:06 |
| fungi | i would normally have renewed expiring certs, but given some of the ones i wanted to sync up weren't expiring for many more months, namecheap wasn't giving me a renew option to refresh their signature on the old csr | 18:06 |
| fungi | so i just generated new csrs for all our certs in one go | 18:07 |
| fungi | because, lots of repetitive clicking on things some ~20 times | 18:07 |
| jeblair | fungi: that makes sense. i don't understand why that wasn't seamless though. | 18:07 |
| *** rybridges has joined #openstack-infra | 18:08 | |
| jeblair | nope. laptop works as expected. i'm very confused. | 18:08 |
| fungi | in the past i've noticed my browser hanging onto outdated cert material when it remained valid and not update without a force reload or cache clear... but hadn't ever witnessed it persist through an expiration | 18:08 |
| openstackgerrit | Merged openstack/os-client-config master: Updated from global requirements https://review.openstack.org/491294 | 18:08 |
| jeblair | i had just restarted the browser too; so even more reason not to have anything cached. | 18:09 |
| SpamapS | I wonder if we tried to diagram all the layers of caching between your eyeballs and storyboard if we'd go completely insane. | 18:10 |
| jeblair | okay, i may just have to chalk this one up to 'writing web browsers is hard'. | 18:10 |
| jeblair | SpamapS: the ones in my brain are *particularly weird* (the fact that our eyeballs *have a cache* is amazing) | 18:10 |
| *** EricGonczer_ has joined #openstack-infra | 18:11 | |
| *** sshnaidm has joined #openstack-infra | 18:13 | |
| *** EricGonc_ has joined #openstack-infra | 18:14 | |
| fungi | yep, persistence of vision is wild | 18:14 |
| *** EricGonczer_ has quit IRC | 18:15 | |
| *** annegentle has quit IRC | 18:17 | |
| *** d0ugal has quit IRC | 18:19 | |
| *** eranrom has joined #openstack-infra | 18:21 | |
| *** felipemonteiro has joined #openstack-infra | 18:23 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:23 | |
| *** jamesdenton has quit IRC | 18:26 | |
| *** felipemonteiro has quit IRC | 18:28 | |
| *** felipemonteiro_ has quit IRC | 18:28 | |
| *** felipemonteiro__ has joined #openstack-infra | 18:28 | |
| *** felipemonteiro has joined #openstack-infra | 18:29 | |
| *** psachin has quit IRC | 18:30 | |
| *** jamesdenton has joined #openstack-infra | 18:31 | |
| *** felipemonteiro__ has quit IRC | 18:32 | |
| *** sshnaidm is now known as sshnaidm|off | 18:35 | |
| *** annegentle has joined #openstack-infra | 18:39 | |
| SpamapS | jeblair: yeah, whoever or whatever designed our internal cache with mutable storage was either hilarious or lacked sentience. | 18:40 |
| *** tnovacik has joined #openstack-infra | 18:42 | |
| fungi | evidence suggests it was for comedic effect | 18:47 |
| mnaser | pabelanger looks like 18 passes, 2 fails in the past 6 hpirs | 18:47 |
| mnaser | hours* | 18:47 |
| mnaser | both fails were non voting jobs | 18:47 |
| pabelanger | mnaser: nice, job runs seem to line up? | 18:49 |
| *** florianf has quit IRC | 18:49 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Bump vexxhost max-servers to 20 https://review.openstack.org/493642 | 18:51 |
| mnaser | fungi pabelanger ^ should help a bit more and i think its going through smoothly | 18:51 |
| pabelanger | \o/ +2 | 18:51 |
| clarkb | approved, I also approved the rubygems proxying change | 18:52 |
| pabelanger | great, thanks | 18:52 |
| *** markvoelker has joined #openstack-infra | 18:53 | |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Skip yum-config-manager on Fedora https://review.openstack.org/493643 | 18:54 |
| clarkb | fungi: pabelanger you have been doing much of the proxy stuff, when do you think would be a good opportunity to clear out the old 3 character apache cache dir entries? | 18:54 |
| clarkb | I put my find command in the comments of my change that switched to two character dirs | 18:55 |
| clarkb | if you want to take a look at the method I was planning to use that would be good too | 18:55 |
| pabelanger | clarkb: I don't have a preference, we could start now or wait until tomorrow. | 18:56 |
| pabelanger | mirror.kna1 is new online today, and hasn't have issues repriming both proxy and afs caches | 18:57 |
| clarkb | pabelanger: I'm thinking earlier is better just to get the proxies as happy as possible before people really start to dig into the week | 18:58 |
| *** jtomasek has quit IRC | 18:58 | |
| clarkb | but fungi may have other input based on stuff that went on over the weekend | 18:58 |
| pabelanger | Ya, if fungi is okay with it, we could start now | 18:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Add more content to zuul v3 migration https://review.openstack.org/489691 | 19:02 |
| openstackgerrit | Merged openstack-infra/system-config master: Add rubygems.org to reverse proxy cache https://review.openstack.org/493316 | 19:03 |
| jeblair | AJaeger: ^ fixed those nits | 19:03 |
| *** adisky__ has quit IRC | 19:05 | |
| fungi | clarkb: pabelanger: looking now | 19:09 |
| *** eranrom has quit IRC | 19:10 | |
| fungi | clarkb: worried that the more data we blow away in those caches, the bigger hit we're going to cause (particularly to tarballs.o.o) | 19:11 |
| fungi | maybe we should at least wait until activity dies down for the day so that it's less likely they'll all slam stuff at once? | 19:11 |
| *** felipemonteiro has quit IRC | 19:12 | |
| openstackgerrit | Merged openstack-infra/project-config master: Bump vexxhost max-servers to 20 https://review.openstack.org/493642 | 19:12 |
| clarkb | fungi: ya or we could try to rate limit it with sleeps between deletes or something | 19:12 |
| fungi | a combination of the two might be in order | 19:13 |
| *** felipemonteiro has joined #openstack-infra | 19:13 | |
| *** kjackal_ has joined #openstack-infra | 19:16 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add settings for OVB in RDO Cloud https://review.openstack.org/480950 | 19:16 |
| clarkb | fungi: if we did a one second sleep between deletes it should take about 2-2.5 days to delete everything | 19:21 |
| clarkb | `find /var/cache/apache2/proxy -type d -name '???' -exec rm -rf {} \; -exec sleep 1 \;` something like that | 19:22 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Allow launcher to stop quicker when asked https://review.openstack.org/492219 | 19:23 |
| *** jamesden_ has joined #openstack-infra | 19:25 | |
| clarkb | fungi: do you think that will be a sufficient spread of the cache invalidation? | 19:25 |
| *** slaweq has joined #openstack-infra | 19:25 | |
| *** eranrom has joined #openstack-infra | 19:25 | |
| fungi | clarkb: oh, no i meant space out starting mass deletion on each mirror | 19:25 |
| fungi | by like maybe 30 minutes between each | 19:26 |
| *** jamesdenton has quit IRC | 19:26 | |
| fungi | or even less | 19:26 |
| clarkb | ah | 19:26 |
| clarkb | ya that is easily doable | 19:26 |
| clarkb | according to zuul status graphs we should be mostly caught up on the job side in a few hours, I can kick off the firstone once we see we aren't running behind consistently | 19:27 |
| fungi | just worried that _if_ we manage to rapidly remove content from all mirrors simultaneously, and we're doing it while still running at max capacity, we could ddos tarballs.o.o and possibly other sites serving large files | 19:27 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add settings for OVB in RDO Cloud https://review.openstack.org/480950 | 19:27 |
| clarkb | ya | 19:27 |
| fungi | especially since doing it now, we'll effectively remove most of the cache from each mirror (as opposed to if we waited days/weeks for a lot of files to get updated and cycled out) | 19:28 |
| clarkb | yup | 19:29 |
| clarkb | though its possible cycling out won't/can't happen until we put this in place | 19:29 |
| * clarkb checks how far off of 50GB other clouds are | 19:29 | |
| clarkb | dfw is at 76GB and has a 5.5 hour htcacheclean running | 19:30 |
| clarkb | iad is at 71GB and has a half hour htcacheclean running | 19:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Publish placement-api-ref https://review.openstack.org/491860 | 19:35 |
| *** ldnunes has quit IRC | 19:39 | |
| *** mat128 has joined #openstack-infra | 19:39 | |
| *** jbadiapa has quit IRC | 19:40 | |
| *** mat128 has quit IRC | 19:43 | |
| *** mat128 has joined #openstack-infra | 19:44 | |
| *** armax has joined #openstack-infra | 19:46 | |
| *** mat128 has quit IRC | 19:46 | |
| clarkb | ok I've confirmed that on ubuntu at least acpi-support package makes our VMs shutdown cleanly. I'll work on adding a patch to infra package needs for that once I figure out what the package is for other distros | 19:47 |
| *** mat128 has joined #openstack-infra | 19:47 | |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config master: Add tripleo scenario002 to Cinder experimental https://review.openstack.org/493655 | 19:49 |
| *** felipemonteiro_ has joined #openstack-infra | 19:50 | |
| *** ldnunes has joined #openstack-infra | 19:51 | |
| pabelanger | fungi: clarkb: any more thoughts on the glean --skip-dns change: https://review.openstack.org/493327 | 19:52 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add settings for OVB in RDO Cloud https://review.openstack.org/480950 | 19:53 |
| *** mat128 has quit IRC | 19:53 | |
| *** felipemonteiro has quit IRC | 19:54 | |
| *** skelso has joined #openstack-infra | 19:54 | |
| clarkb | pabelanger: is there a change that shows how we would consume that? | 19:56 |
| clarkb | thats the systemd stuff right? | 19:56 |
| pabelanger | clarkb: not yet, I can write up how it would look | 19:57 |
| clarkb | I've +2'd the change was mostly curious how you wanted to get that in because simple init element in dib likely can't hardcode that right? | 19:58 |
| fungi | i don't suppose we inject glean from source in the nodepool integration job | 19:58 |
| clarkb | (since it needs to be flexible and work with generic image build case) | 19:58 |
| clarkb | fungi: ya thats what the -src jobs are | 19:58 |
| fungi | ahh | 19:58 |
| fungi | so in theory... | 19:58 |
| fungi | we could exercise this implementation with a depends-on without merging/releasing if we wanted to iron out any issues | 19:59 |
| clarkb | ya | 20:00 |
| clarkb | though its a devstack install we are booting into which won't boot nodes with unbound iirc | 20:00 |
| pabelanger | Ya, I was going to land the change in project-config, over diskimage-builder directly. We don't build our elements today | 20:00 |
| clarkb | however they should mostlywork for testing without dns | 20:00 |
| fungi | oh, got it | 20:02 |
| *** Swami has quit IRC | 20:05 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Add acpid to infra images https://review.openstack.org/493658 | 20:05 |
| clarkb | that is likely to be just a convenience change But it may be related to why we are leaking instances if nova isn't able to be as forceful deleting those instances as necssary | 20:06 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add settings for OVB in RDO Cloud https://review.openstack.org/480950 | 20:07 |
| *** slaweq has quit IRC | 20:08 | |
| *** e0ne has joined #openstack-infra | 20:09 | |
| *** eranrom has quit IRC | 20:10 | |
| *** mpjetta_ has joined #openstack-infra | 20:10 | |
| pabelanger | clarkb: we basically create override.conf files per distro, since glean.sh is installed in different locations: http://paste.openstack.org/show/618337/ | 20:11 |
| pabelanger | then systems glean@.service will get setup properly to use override.conf | 20:11 |
| clarkb | why is glean.sh in different locations? | 20:12 |
| clarkb | and we'd do that in an infra element? | 20:12 |
| *** eranrom has joined #openstack-infra | 20:12 | |
| *** d0ugal has joined #openstack-infra | 20:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Add more content to zuul v3 migration https://review.openstack.org/489691 | 20:13 |
| pabelanger | clarkb: because pip and datafiles | 20:13 |
| pabelanger | /usr/local/bin/glean.sh vs /usr/bin/glean.sh | 20:13 |
| pabelanger | distro specific paths | 20:14 |
| clarkb | I see | 20:14 |
| *** mpjetta has quit IRC | 20:14 | |
| *** mpjetta_ is now known as mpjetta | 20:14 | |
| pabelanger | clarkb: we could hack up systemd and do something like ExecStart=/bin/sh -c "$(which glean.sh) ..." | 20:15 |
| pabelanger | hacky because ExecStart commands require / as the first char | 20:16 |
| clarkb | ya and which isn't everywhere | 20:16 |
| clarkb | not worth it was mostly just curious why it was needed in the first place | 20:16 |
| *** eharney has quit IRC | 20:16 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_delayed_repo_init https://review.openstack.org/493659 | 20:21 |
| *** lewo has quit IRC | 20:23 | |
| *** lewo has joined #openstack-infra | 20:24 | |
| *** felipemonteiro_ has quit IRC | 20:27 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:27 | |
| *** eranrom has quit IRC | 20:27 | |
| *** felipemonteiro__ has joined #openstack-infra | 20:28 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Fix neutron extension name https://review.openstack.org/493661 | 20:29 |
| Shrews | mordred: ^^^ | 20:29 |
| *** e0ne_ has joined #openstack-infra | 20:29 | |
| *** e0ne has quit IRC | 20:30 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add settings for OVB in RDO Cloud https://review.openstack.org/480950 | 20:30 |
| clarkb | 47GB used in la1 citycloud mirror cache | 20:32 |
| *** felipemonteiro_ has quit IRC | 20:32 | |
| *** jamesmcarthur has quit IRC | 20:33 | |
| *** jamesmcarthur has joined #openstack-infra | 20:34 | |
| *** sekelso has joined #openstack-infra | 20:34 | |
| *** skelso has quit IRC | 20:35 | |
| fungi | kmARC: when you're ready to talk about the upstream-institute ci jobs, if what you're stuck on is having jobs that create container images and upload them to tarballs.o.o, i'm pretty sure there are already multiple projects doing things like that (e.g. kolla) which could provide inspiration | 20:36 |
| *** e0ne_ has quit IRC | 20:37 | |
| kmARC | okay, so before we try to shortcut and go to examples, I wanna give some hints why this is the way we chose | 20:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create glean@.service.d/override.conf https://review.openstack.org/493665 | 20:37 |
| *** Apoorva_ has joined #openstack-infra | 20:38 | |
| pabelanger | clarkb: glean@.service.d file would look something like ^ | 20:38 |
| *** jamesmcarthur has quit IRC | 20:38 | |
| pabelanger | still to be tested | 20:38 |
| pabelanger | tested in, build image and validate | 20:38 |
| *** Apoorva has quit IRC | 20:38 | |
| kmARC | We want a very, very small image that people can distribute quickly on the training. Currently the record is around 900MB, and that already contains a devstack installation. How we achieve this, is basically to run a couple bash scripts (easy) on a booted VM and then "defragment" the disk and then compress it with vmware tooling. | 20:40 |
| *** efoley_ has quit IRC | 20:40 | |
| kmARC | that's the disk part | 20:40 |
| fungi | kmARC: so you're saying you need to configure a custom vm image booted within one of our ci job vms? | 20:40 |
| kmARC | the configuration is done by vagrant, it's kinda convenient how we can set up the machine to have the correct configs for macs, windows, linux desktops regarding networking and disk format | 20:41 |
| *** jamesmcarthur has joined #openstack-infra | 20:41 | |
| kmARC | well, yes. As long as that vm can boot up from vmdk without conversion, that should work out of the box | 20:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Zuul v3: rename allow-secrets to post-review https://review.openstack.org/493667 | 20:41 |
| jeblair | fungi, clarkb, mordred, pabelanger: ^ can we instamerge that? | 20:41 |
| fungi | kmARC: also, i didn't know vmware made free/libre open source tools for that. which ones are you using specifically? | 20:41 |
| kmARC | vmware-vdiskimager | 20:43 |
| fungi | kmARC: okay, so to reset this discussion to basics (as i think there may be some unspoken assumptions)... | 20:44 |
| clarkb | jeblair: I've +2'd not sure if you mean force merge or just fast approve re instamerge but I think either is fine if you need to do it | 20:44 |
| AJaeger | Wow, post pipeline is empty, we have idle nodes again ;) | 20:44 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo: make undercloud-container non-voting https://review.openstack.org/493669 | 20:44 |
| EmilienM | weshay, pabelanger, dprince ^ | 20:44 |
| jeblair | clarkb: fast merge | 20:44 |
| jeblair | clarkb: ha, i mean fast approve | 20:44 |
| AJaeger | jeblair: +2a | 20:44 |
| jeblair | clarkb: (i answered your request for clarification with more ambiguity!) | 20:44 |
| fungi | kmARC: our jobs run via ansible over ssh in a single-use server booted from a generic but custom build of your choice of a number of popular linux distros | 20:45 |
| *** baoli_ has joined #openstack-infra | 20:45 | |
| fungi | kmARC: you can install anything you want on that server, run whatever you need to on that server | 20:45 |
| kmARC | am I able to boot up a virtualbox VM on it...? | 20:45 |
| fungi | kmARC: the fundamental limitations of what you can assume about the server are listed here: https://docs.openstack.org/infra/manual/testing.html | 20:46 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix neutron extension name https://review.openstack.org/493661 | 20:46 |
| fungi | kmARC: i don't know if anybody's tried to run virtualbox under those conditions, but it's worth a try? | 20:46 |
| *** baoli has quit IRC | 20:47 | |
| kmARC | this doc says: Nested virt is not available in all clouds. | 20:47 |
| kmARC | That might be a problem | 20:47 |
| clarkb | virtualbox doesn't need nested virt | 20:48 |
| fungi | kmARC: we recommend operating in containers or a chroot instead if you need additional layers. there are some jobs which perform basic functions within a nested vm, but that page is right you can't really rely on nested virtualization acceleration | 20:48 |
| clarkb | at least it didn't way back when | 20:48 |
| fungi | you can still boot nested virtual machines, there's just no hardware acceleration to passthrough operations to the underlying hypervisor | 20:49 |
| mordred | yah. so it tends to be SLOW | 20:49 |
| kmARC | clarkb: hmm... I'll try it. DO you have a tenant where I could play around for a couple days? | 20:49 |
| *** baoli_ has quit IRC | 20:51 | |
| *** gouthamr has quit IRC | 20:51 | |
| *** baoli has joined #openstack-infra | 20:51 | |
| fungi | kmARC: the agreements with our donors don't really cover that, but any single provider donated to us is unlikely to provide you with a complete picture. you might see if you can get the basics to work on a qemu-booted virtual machine with nested virt passthrough disabled and then put together a limited ci job once you confirm whether or not it functions at all | 20:52 |
| kmARC | yeah, as a fallback yes. It would've been just more convenient :-) | 20:52 |
| *** rbrndt_ has joined #openstack-infra | 20:53 | |
| *** rbrndt has quit IRC | 20:53 | |
| fungi | i don't know the state of trystack.org at this stage to know whether or not it's something you could try on. alternatively i heard ovh has a free trial? | 20:53 |
| fungi | i don't recall the details, nor am i sure they're still running it, but maybe that's an option to consider | 20:54 |
| clarkb | I personally just run VMs on my local desktop | 20:55 |
| kmARC | clarkb: I have no means right now to provision an ubuntu VM within 1 minute on my desktop | 20:56 |
| kmARC | with qemu* | 20:56 |
| kmARC | with virtualbox, yes I do | 20:56 |
| mnaser | f | 20:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Zuul v3: rename allow-secrets to post-review https://review.openstack.org/493667 | 20:59 |
| *** spzala has quit IRC | 21:00 | |
| mnaser | does nodepool get reconfigured right after project-config stuff merge? | 21:00 |
| openstackgerrit | Merged openstack-infra/glean master: Add --skip-dns option https://review.openstack.org/493327 | 21:00 |
| pabelanger | mnaser: usually takes 15-30mins | 21:00 |
| mnaser | pabelanger looking at openstack tenant i see only 10 servers even tho we merged https://review.openstack.org/#/c/493642/ 2 hours ago | 21:01 |
| clarkb | kmARC: not all of our cloud instances are qemu/kvm (though my local ones are) | 21:01 |
| pabelanger | mnaser: ya, we are configured for 20 now but don't have the demand to launch more | 21:02 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-vexxhost | 21:02 |
| mnaser | pabelanger oh forogt about grafana | 21:02 |
| *** dmsimard|off has quit IRC | 21:03 | |
| *** dprince has quit IRC | 21:03 | |
| clarkb | pabelanger: fungi with demand fallen off is now a good time to remove the old cache dirs on a mirror? | 21:03 |
| clarkb | I'm thinking I would start with rax-dfw since it is fairly large and adjacent to eg tarballs | 21:04 |
| *** trown is now known as trown|outtypewww | 21:04 | |
| fungi | clarkb: i'd say so, yes | 21:04 |
| pabelanger | clarkb: ++ | 21:05 |
| *** thorst has quit IRC | 21:05 | |
| fungi | it seems to have finally gotten to all the periodic jobs from ~15 hours ago | 21:05 |
| clarkb | ok will run in a root screen on dfw mirror | 21:06 |
| openstackgerrit | Merged openstack-infra/shade master: Add functional tests for Neutron QoS policies and rules https://review.openstack.org/491248 | 21:06 |
| *** jamesmcarthur has quit IRC | 21:06 | |
| *** ilpianista_ has quit IRC | 21:07 | |
| *** yamamoto has joined #openstack-infra | 21:08 | |
| *** masayukig[m] has quit IRC | 21:08 | |
| clarkb | looks like it will delete fairly slwoly anyways | 21:08 |
| fungi | probably. tons of inode operations over a sluggish/oversibscribed iscsi network | 21:09 |
| *** rbrndt_ has quit IRC | 21:10 | |
| *** jamesmcarthur has joined #openstack-infra | 21:11 | |
| *** dangers_away has quit IRC | 21:12 | |
| *** manjeets has quit IRC | 21:14 | |
| *** ldnunes has quit IRC | 21:15 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Update NODEPOOL_RUGYGEMS_PROXY / add bundle client support https://review.openstack.org/493681 | 21:16 |
| *** tushar_ has quit IRC | 21:16 | |
| *** eharney has joined #openstack-infra | 21:17 | |
| pabelanger | clarkb: fungi: Just confirmed bundle / gem install on vexxhost working as expected ^ We finally have gems being cached | 21:19 |
| *** yamamoto has quit IRC | 21:20 | |
| *** jamesmcarthur has quit IRC | 21:21 | |
| fungi | woo! excellent work | 21:21 |
| *** rbrndt has joined #openstack-infra | 21:21 | |
| *** jamesmcarthur has joined #openstack-infra | 21:21 | |
| *** dangers has joined #openstack-infra | 21:23 | |
| *** jamesmcarthur has quit IRC | 21:23 | |
| pabelanger | clarkb: care to review / +3 https://review.openstack.org/492671/ our first secrets for zuulv3. Would like to start testing that on zuulv3.o.o | 21:25 |
| clarkb | pabelanger: it was encrypted with the gerrit ssh key? | 21:26 |
| pabelanger | clarkb: nothing else, just the private key | 21:26 |
| clarkb | is that because zuul already has access to tht key for ssh to gerrit? | 21:27 |
| *** jamesmcarthur has joined #openstack-infra | 21:27 | |
| pabelanger | clarkb: oh, you mean how I encrypted it? | 21:27 |
| *** sekelso has quit IRC | 21:28 | |
| clarkb | yes | 21:28 |
| *** sekelso has joined #openstack-infra | 21:28 | |
| clarkb | I'm trying to understand why we'd use a gerrit key for something mostly unrelated | 21:28 |
| pabelanger | clarkb: I used tools/encrypt_secret.py from zuul, and it pulled down the public key via HTTP from zuulv3.o.o | 21:28 |
| pabelanger | gerrit is the project | 21:29 |
| pabelanger | err, tennant? | 21:29 |
| clarkb | ah ok I read that as "we used the key that we ssh into gerrit with" | 21:29 |
| pabelanger | ya, source gerrit, project openstack-infra/project-config from zuulv3.o.o | 21:30 |
| pabelanger | via HTTP | 21:30 |
| *** thorst has joined #openstack-infra | 21:30 | |
| *** dmsimard|off has joined #openstack-infra | 21:30 | |
| *** dave-mcc_ has joined #openstack-infra | 21:31 | |
| clarkb | pabelanger: and is that a new ssh key or are we reusing the existing one? | 21:31 |
| pabelanger | that is our production SSH key we are using today | 21:32 |
| clarkb | thinking replacing the existing one may be more painful than a new one if zuul somehow discloses that key unexpectedly | 21:32 |
| *** sbezverk has quit IRC | 21:32 | |
| clarkb | fungi: is ^ overly paranoid? | 21:32 |
| clarkb | pabelanger: is that the same key we use to ssh into all the test nodes? | 21:32 |
| pabelanger | we could create a new zuul user on static.o.o with new keys if needed | 21:33 |
| clarkb | pabelanger: doesn't have to be a new user, just a second authorized_key for jenkins | 21:33 |
| *** sree has joined #openstack-infra | 21:33 | |
| pabelanger | clarkb: that is the same key we use on zuul-launchers to upload to tarballs.o.o. No test nodes should have it today | 21:33 |
| *** dave-mccowan has quit IRC | 21:33 | |
| clarkb | pabelanger: just thinking that limits exposure to the logs server/static and not everything else that key can hit | 21:34 |
| *** EricGonc_ has quit IRC | 21:34 | |
| *** thorst has quit IRC | 21:34 | |
| pabelanger | sure, I'll update system-config and add support for new keys | 21:34 |
| *** jamesmcarthur has quit IRC | 21:34 | |
| clarkb | pabelanger: I'd be curious what fungi think since he is typically quite paranoid, if he isn't worried then I think its fine as is | 21:35 |
| *** jamesmcarthur has joined #openstack-infra | 21:35 | |
| fungi | parsing... the suggestion is to have the zuul launchers use two different keys, one for bootstrapping their access to test nodes and another for uploading files to places like static.o.o? | 21:35 |
| fungi | sorry, not launchers now. executors | 21:36 |
| fungi | v3speak | 21:36 |
| *** thorst has joined #openstack-infra | 21:36 | |
| clarkb | fungi: no, suggesting that zuulv2.5 and zuulv3 use different ssh keys to get to places like static.o.o to limit cost of exposure should zuulv3 expose that private key unexpectedly while in bootstrapping/development | 21:36 |
| clarkb | this way we won't have to rekey everything, just a subset | 21:36 |
| *** baoli has quit IRC | 21:37 | |
| fungi | not a terrible idea, though we're only a few weeks from dumping tat server anyway so it may not be worth the added effort | 21:37 |
| clarkb | ew currently do it for access to gerrit at least | 21:37 |
| clarkb | (they use different users with different keys irght?) | 21:37 |
| fungi | depends on how much extra work it is to add one more | 21:38 |
| pabelanger | ya | 21:38 |
| *** sree has quit IRC | 21:38 | |
| *** thorst has quit IRC | 21:39 | |
| pabelanger | jobs don't directly expose zuul user ssh private key for review.o.o, they'd have to break out of ansible then bwrap to access it | 21:39 |
| *** masayukig[m] has joined #openstack-infra | 21:39 | |
| inc0 | hey, still on track to be able to do zuulv3 jobs soon? | 21:39 |
| *** EricGonczer_ has joined #openstack-infra | 21:39 | |
| pabelanger | which, we are saying is okay right now | 21:39 |
| clarkb | pabelanger: I'd be more worried about something getting logged as a fact or that type of disclosure unexpectedly | 21:39 |
| pabelanger | clarkb: Ya, same. This is going to be the first time we try secrets, so we should be prepared to leak something | 21:40 |
| clarkb | (basiclly nothing malicious or intentional, just new system growing pains | 21:40 |
| *** lin_yang has joined #openstack-infra | 21:41 | |
| *** dangers has quit IRC | 21:41 | |
| pabelanger | so, if we generate new key and leak, somebody will get access to static.o.o. Our original key will still be safe however | 21:41 |
| clarkb | right | 21:41 |
| pabelanger | we'd rekey, maybe keep tarballs.o.o up? | 21:41 |
| pabelanger | err, static.o.o | 21:41 |
| openstackgerrit | Merged openstack-infra/shade master: Consolidate client version checks in an utility method https://review.openstack.org/493598 | 21:42 |
| clarkb | right so I think question is is there value in that? fungi ^ ? | 21:42 |
| *** rbrndt has quit IRC | 21:43 | |
| fungi | basically what using different keys for access to static.o.o would give us is the ability to avoid replacing the zuul v2 ssh key if zuul v3 leaks the one it's using | 21:43 |
| fungi | so, right, not really much pain saved | 21:43 |
| clarkb | pabelanger: and this is not the key used to ssh to gerrit right? | 21:46 |
| pabelanger | what about just using zuulv3-dev.o.o as a upload server until we are confident about not leaking something | 21:46 |
| *** jamesmcarthur has quit IRC | 21:46 | |
| pabelanger | we still have that server online, doing nothing | 21:46 |
| fungi | seems safe enough, sure | 21:46 |
| clarkb | pabelanger: ya we could start with that really quick then switch the key and fqdn over | 21:46 |
| clarkb | that sounds straightforward. | 21:46 |
| pabelanger | k, I'll WIP and make changes | 21:47 |
| fungi | agreed, v3 makes it easier! | 21:47 |
| jeblair | wait what? | 21:47 |
| jeblair | i see we're talking about 492671 which i just approved | 21:48 |
| jeblair | so i have removed my +w | 21:48 |
| jeblair | but gimme a second to review this backlog | 21:48 |
| jeblair | pabelanger, fungi, clarkb: i'm fine in theory with us adding a new key to static.o.o, however, it's very much not straightforward | 21:50 |
| jeblair | i was going to do that, and then realized the puppet cruft went all the way down to puppet-jenkins | 21:51 |
| *** rbrndt has joined #openstack-infra | 21:51 | |
| jeblair | and we discussed this and decided it was okay to keep using the current key | 21:51 |
| jlvillal | Anyone know if there is some group code to book the hotel for PTG? | 21:51 |
| fungi | jeblair: yeah, we came to a similar conclusion here | 21:52 |
| *** jamesmcarthur has joined #openstack-infra | 21:52 | |
| *** ilpianista_ has joined #openstack-infra | 21:52 | |
| fungi | pabelanger suggested testing publishing to a dev server first temporarily | 21:52 |
| jeblair | :( | 21:52 |
| clarkb | jlvillal: my understanding is that it is fully booked | 21:52 |
| fungi | and then switching to static.o.o after | 21:52 |
| jlvillal | clarkb: Ah thanks. And :( | 21:52 |
| jeblair | fungi, pabelanger: if we're worried about secrets working at all, maybe we can just merge a quick test secret? | 21:53 |
| clarkb | jeblair: ya I'm mostly just wanting to make sure it works without using production data first | 21:53 |
| fungi | jlvillal: i've been told by the organizers that there are still rooms available, but they're sold out of the group discount rate rooms as of a week or so ago | 21:53 |
| clarkb | doesn't necessarily have to be an ssh key for logs uploads | 21:53 |
| jlvillal | fungi: Thanks. | 21:54 |
| clarkb | we print facts in some jobs for example, will that expose the key? what about ara? | 21:54 |
| fungi | jlvillal: so have to book through the hotel at the regular rate now | 21:54 |
| clarkb | mostly just thinking sanity check beforehand may be good | 21:54 |
| *** xyang1 has quit IRC | 21:54 | |
| pabelanger | creating secret: foobar first should be fine, then templating that into /tmp/blah should be the same steps. We can then audit logs and look for leak | 21:54 |
| jeblair | okay. that should actually be a unit test. | 21:55 |
| fungi | jlvillal: good news is, the conference hotel seems to have been a much more popular choice this time, so maybe next ptg they can feel more comfortable about increasing the room block | 21:55 |
| pabelanger | jeblair: completely agree | 21:55 |
| fungi | jeblair: i thought that already was a unit test? | 21:55 |
| fungi | or am i thinking of a different one? | 21:55 |
| jeblair | fungi: beats me | 21:55 |
| jeblair | but if there are any things about this we are inclined to test, i highly recommend we just do it as a unit test | 21:56 |
| jeblair | because then it's valid for the future, not just this one time. :) | 21:56 |
| *** ihrachys has quit IRC | 21:56 | |
| fungi | i'm cool with that | 21:56 |
| jeblair | but in general, i'm kinda pushing the idea that the jobs we're writing are production jobs. we're way too close to the ptg to be thinking about testing the waters on really basic stuff. | 21:57 |
| *** yamamoto has joined #openstack-infra | 21:57 | |
| jeblair | if that makes us feel rushed, let's call off the ptg cutover. | 21:57 |
| *** spzala has joined #openstack-infra | 21:58 | |
| *** spzala has quit IRC | 21:58 | |
| *** felipemonteiro__ has quit IRC | 21:59 | |
| *** spzala has joined #openstack-infra | 21:59 | |
| *** spzala has quit IRC | 21:59 | |
| jeblair | segue to zuul meeting time in #openstack-meeting-alt | 22:00 |
| clarkb | I was mostly just trying to understand what the key is already used for and whether it was appropriate to use it there. We already use a different key for gerrit for example | 22:01 |
| clarkb | and have reverted use of the gearman CA for example | 22:01 |
| clarkb | I don't think its a matter of being rushed so much as understanding what the change means | 22:01 |
| *** rbrndt has quit IRC | 22:04 | |
| *** sekelso has quit IRC | 22:08 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/devstack-gate master: Update grenade settings for stable/pike https://review.openstack.org/493057 | 22:08 |
| *** dteselkin has quit IRC | 22:11 | |
| *** dave-mcc_ has quit IRC | 22:11 | |
| *** katyafervent_awa has quit IRC | 22:12 | |
| *** katyafervent_awa has joined #openstack-infra | 22:15 | |
| *** dteselkin has joined #openstack-infra | 22:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove openstack-publish-tarball base job https://review.openstack.org/492714 | 22:16 |
| *** tnovacik has quit IRC | 22:17 | |
| *** rbrndt has joined #openstack-infra | 22:19 | |
| *** gibi has quit IRC | 22:23 | |
| *** rbrndt has quit IRC | 22:26 | |
| *** jamesmcarthur has quit IRC | 22:29 | |
| *** EricGonczer_ has quit IRC | 22:35 | |
| *** gouthamr has joined #openstack-infra | 22:39 | |
| *** thorst has joined #openstack-infra | 22:39 | |
| *** vhosakot has quit IRC | 22:43 | |
| *** thorst has quit IRC | 22:44 | |
| *** dave-mccowan has joined #openstack-infra | 22:48 | |
| *** gibi has joined #openstack-infra | 22:51 | |
| *** dave-mccowan has quit IRC | 22:51 | |
| *** dhill_ has quit IRC | 22:51 | |
| *** dhill_ has joined #openstack-infra | 22:51 | |
| clarkb | dfw mirror just finished cleaning up its dirs | 22:58 |
| clarkb | fungi: pabelanger I'm going to run on iad next | 22:59 |
| *** jamesmcarthur has joined #openstack-infra | 22:59 | |
| clarkb | fungi: do you want me to delay before starting there? | 22:59 |
| clarkb | I think that the rm's were slow enough that we shouldn't have to worry too much | 22:59 |
| fungi | clarkb: cool, can probably do some of them in parallel, just offset the start times a little to avoid necessarily impacting the same stuff all at once (though i guess that's not likely anyway given random-ish layouts and the time it takes to complete) | 22:59 |
| clarkb | ya I think it should be fairly well distributed just due to timing | 23:00 |
| fungi | also, zuul is caught up and we're only using about half our capacity now | 23:01 |
| fungi | and it will probably only get slower still until ~0600z when periodic jobs start again | 23:01 |
| *** rcernin has quit IRC | 23:02 | |
| *** marst has quit IRC | 23:03 | |
| *** jamesmcarthur has quit IRC | 23:03 | |
| pabelanger | clarkb: fungi: jeblair: Where did we stand on production SSH for jenkins user on static.o.o? I confirmed we do have a unit test in place to test secrets, but unsure how to audit of we leak the key in logs (looked manually didn't see anything). And following playbooks no longer have access to secrets.yaml files. So pretty confident we shouldn't leak by default | 23:05 |
| *** jamesmcarthur has joined #openstack-infra | 23:07 | |
| clarkb | iad deletion in progress. | 23:07 |
| mordred | pabelanger: maybe a test job that adds a secret we dont' care about and uses it (but doens't echo it anywhere) - that way we can look at the various logs from a live job? | 23:07 |
| clarkb | jeblair: is the mtl01 mirror being switched soon or should I clean up the cache on the old proxy? | 23:07 |
| pabelanger | mordred: ya, I can quickly write that | 23:08 |
| clarkb | kna1 is done by virtue of being rebuilt | 23:08 |
| clarkb | going to do la1 now | 23:08 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_delayed_repo_init https://review.openstack.org/493659 | 23:13 |
| *** jamesmcarthur has quit IRC | 23:13 | |
| clarkb | pabelanger: how has sto2 been? | 23:17 |
| *** pbourke has quit IRC | 23:18 | |
| *** pbourke has joined #openstack-infra | 23:20 | |
| *** tosky has quit IRC | 23:20 | |
| fungi | pabelanger: it sounded like the suggestion was to add a unit test that uses a fixture config with a secret in it, does something and confirms we didn't leak it | 23:21 |
| fungi | that way we're covered for more than just this once | 23:22 |
| *** dteselkin has quit IRC | 23:22 | |
| *** katyafervent_awa has quit IRC | 23:22 | |
| *** katyafervent_awa has joined #openstack-infra | 23:24 | |
| clarkb | la1 is done, now doing sto2 | 23:26 |
| *** dizquierdo has joined #openstack-infra | 23:27 | |
| *** dteselkin has joined #openstack-infra | 23:27 | |
| *** mattmceuen has quit IRC | 23:30 | |
| *** spzala_ has joined #openstack-infra | 23:31 | |
| *** dangers_away has joined #openstack-infra | 23:31 | |
| *** sree has joined #openstack-infra | 23:34 | |
| pabelanger | clarkb: good so far, I haven't see anything fail just yet | 23:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] LVM support for dib-block-device https://review.openstack.org/472065 | 23:36 |
| openstackgerrit | Jamey McCabe proposed openstack-infra/irc-meetings master: Adding new meeting for LCOO WG https://review.openstack.org/492220 | 23:37 |
| *** sree has quit IRC | 23:39 | |
| *** pbourke has quit IRC | 23:51 | |
| *** dhill_ has quit IRC | 23:51 | |
| *** gibi has quit IRC | 23:51 | |
| *** gouthamr has quit IRC | 23:51 | |
| *** Apoorva_ has quit IRC | 23:51 | |
| *** lewo has quit IRC | 23:51 | |
| *** jamesden_ has quit IRC | 23:51 | |
| *** annegentle has quit IRC | 23:51 | |
| *** yamahata has quit IRC | 23:51 | |
| *** caphrim007 has quit IRC | 23:51 | |
| *** krtaylor has quit IRC | 23:51 | |
| *** rlandy has quit IRC | 23:51 | |
| *** AJaeger has quit IRC | 23:51 | |
| *** Dinesh_Bhor has quit IRC | 23:51 | |
| *** aeng has quit IRC | 23:51 | |
| *** panda has quit IRC | 23:51 | |
| *** Jeffrey4l has quit IRC | 23:51 | |
| *** mandre has quit IRC | 23:51 | |
| *** makowals has quit IRC | 23:51 | |
| *** kuromagi has quit IRC | 23:51 | |
| *** calebb has quit IRC | 23:51 | |
| *** lin_yang has quit IRC | 23:51 | |
| *** kjackal_ has quit IRC | 23:51 | |
| *** Guest80741 has quit IRC | 23:51 | |
| *** arturb has quit IRC | 23:51 | |
| *** spligak has quit IRC | 23:51 | |
| *** adriant has quit IRC | 23:51 | |
| *** mjblack has quit IRC | 23:51 | |
| *** bswartz has quit IRC | 23:51 | |
| *** jmorgan1 has quit IRC | 23:51 | |
| *** odyssey4me has quit IRC | 23:51 | |
| *** stevebaker has quit IRC | 23:51 | |
| *** beisner has quit IRC | 23:51 | |
| *** robled has quit IRC | 23:51 | |
| *** reed has quit IRC | 23:51 | |
| *** thingee has quit IRC | 23:51 | |
| *** gorba_ has quit IRC | 23:51 | |
| *** cburgess has quit IRC | 23:51 | |
| *** kong has quit IRC | 23:51 | |
| *** Shrews has quit IRC | 23:51 | |
| *** antonym has quit IRC | 23:51 | |
| *** masayukig[m] has quit IRC | 23:51 | |
| *** d0ugal has quit IRC | 23:51 | |
| *** Julien-zte has quit IRC | 23:51 | |
| *** david-lyle has quit IRC | 23:51 | |
| *** akscram has quit IRC | 23:51 | |
| *** peterlisak has quit IRC | 23:51 | |
| *** pcrews has quit IRC | 23:51 | |
| *** tinwood has quit IRC | 23:51 | |
| *** niska has quit IRC | 23:51 | |
| *** fdegir has quit IRC | 23:51 | |
| *** benj_ has quit IRC | 23:51 | |
| *** Adri2000 has quit IRC | 23:51 | |
| *** larainema has quit IRC | 23:51 | |
| *** dutsmoc has quit IRC | 23:51 | |
| *** asettle has quit IRC | 23:51 | |
| *** cloudnull has quit IRC | 23:51 | |
| *** mnencia has quit IRC | 23:51 | |
| *** fmccrthy has quit IRC | 23:51 | |
| *** dims has quit IRC | 23:51 | |
| *** seongsoocho has quit IRC | 23:51 | |
| *** uberjay has quit IRC | 23:51 | |
| *** wfoster has quit IRC | 23:51 | |
| *** jamielennox has quit IRC | 23:51 | |
| *** yankcrime has quit IRC | 23:51 | |
| *** dgonzalez has quit IRC | 23:51 | |
| *** schang has quit IRC | 23:51 | |
| *** ttx has quit IRC | 23:51 | |
| *** lamt has quit IRC | 23:51 | |
| *** andymccr has quit IRC | 23:51 | |
| *** wznoinsk has quit IRC | 23:51 | |
| *** NikitaKonovalov has quit IRC | 23:51 | |
| *** laurelm has quit IRC | 23:51 | |
| *** vgnbkr has quit IRC | 23:51 | |
| *** peterstac has quit IRC | 23:51 | |
| *** dougshelley66 has quit IRC | 23:51 | |
| *** bkero has quit IRC | 23:51 | |
| *** mmedvede has quit IRC | 23:51 | |
| *** sshnaidm|off has quit IRC | 23:51 | |
| *** rybridges has quit IRC | 23:51 | |
| *** HeOS has quit IRC | 23:51 | |
| *** flaper87 has quit IRC | 23:51 | |
| *** tnarg has quit IRC | 23:51 | |
| *** vipul has quit IRC | 23:51 | |
| *** clayton has quit IRC | 23:51 | |
| *** Hunner has quit IRC | 23:51 | |
| *** olaph has quit IRC | 23:51 | |
| *** isq_ has quit IRC | 23:51 | |
| *** masber has quit IRC | 23:51 | |
| *** dosaboy has quit IRC | 23:51 | |
| *** kaisers2 has quit IRC | 23:51 | |
| *** yuanying has quit IRC | 23:51 | |
| *** bradm has quit IRC | 23:51 | |
| *** rook has quit IRC | 23:51 | |
| *** devananda has quit IRC | 23:51 | |
| *** timothyb89 has quit IRC | 23:51 | |
| *** EmilienM has quit IRC | 23:51 | |
| *** zigo has quit IRC | 23:51 | |
| *** dmsimard|off has quit IRC | 23:51 | |
| *** jkilpatr has quit IRC | 23:51 | |
| *** openstackgerrit has quit IRC | 23:51 | |
| *** jangutter has quit IRC | 23:51 | |
| *** zzzeek has quit IRC | 23:51 | |
| *** markmcd has quit IRC | 23:51 | |
| *** yolanda has quit IRC | 23:51 | |
| *** wendar has quit IRC | 23:51 | |
| *** numans has quit IRC | 23:51 | |
| *** zz_ja has quit IRC | 23:51 | |
| *** hughsaunders has quit IRC | 23:51 | |
| *** csatari has quit IRC | 23:51 | |
| *** DuncanT has quit IRC | 23:51 | |
| *** arxcruz has quit IRC | 23:51 | |
| *** edwarnicke has quit IRC | 23:51 | |
| *** jcook has quit IRC | 23:51 | |
| *** mrmartin has quit IRC | 23:51 | |
| *** lastmikoi has quit IRC | 23:51 | |
| *** dulek has quit IRC | 23:51 | |
| *** esberglu has quit IRC | 23:51 | |
| *** slagle has quit IRC | 23:51 | |
| *** oanson has quit IRC | 23:51 | |
| *** thorre has quit IRC | 23:51 | |
| *** hamzy has quit IRC | 23:51 | |
| *** Nil_ has quit IRC | 23:51 | |
| *** mpranjic has quit IRC | 23:51 | |
| *** aarefiev_afk has quit IRC | 23:51 | |
| *** raissa has quit IRC | 23:51 | |
| *** deduped has quit IRC | 23:51 | |
| *** sambetts|afk has quit IRC | 23:51 | |
| *** rvba has quit IRC | 23:51 | |
| *** Diabelko has quit IRC | 23:51 | |
| *** tristanC has quit IRC | 23:51 | |
| *** rhagarty has quit IRC | 23:51 | |
| *** timburke has quit IRC | 23:51 | |
| *** mika has quit IRC | 23:51 | |
| *** toabctl has quit IRC | 23:51 | |
| *** freerunner has quit IRC | 23:51 | |
| *** Swimingly has quit IRC | 23:51 | |
| *** jeblair has quit IRC | 23:51 | |
| *** tonyb has quit IRC | 23:51 | |
| *** mhayden has quit IRC | 23:51 | |
| *** johnthetubaguy has quit IRC | 23:51 | |
| *** eventingmonkey has quit IRC | 23:51 | |
| *** Keitaro has quit IRC | 23:51 | |
| *** jlk has quit IRC | 23:51 | |
| *** ericyoung has quit IRC | 23:51 | |
| *** zer0c00l_ has quit IRC | 23:51 | |
| *** SotK has quit IRC | 23:51 | |
| *** zeroDivisible has quit IRC | 23:51 | |
| *** bradjones has quit IRC | 23:51 | |
| *** apetrich has quit IRC | 23:51 | |
| *** jistr|off has quit IRC | 23:51 | |
| *** lifeless_ has quit IRC | 23:51 | |
| *** junbo has quit IRC | 23:51 | |
| *** mattoliverau has quit IRC | 23:51 | |
| *** rbergeron has quit IRC | 23:51 | |
| *** niedbalski has quit IRC | 23:51 | |
| *** ggillies has quit IRC | 23:51 | |
| *** fbouliane has quit IRC | 23:51 | |
| *** adam_g has quit IRC | 23:51 | |
| *** pfallenop has quit IRC | 23:51 | |
| *** eglute has quit IRC | 23:51 | |
| *** d34dh0r53 has quit IRC | 23:51 | |
| *** dhellmann has quit IRC | 23:51 | |
| *** jmccrory has quit IRC | 23:51 | |
| *** notmyname has quit IRC | 23:51 | |
| *** lucas-afk has quit IRC | 23:51 | |
| *** sai has quit IRC | 23:51 | |
| *** greghaynes has quit IRC | 23:51 | |
| *** petevg has quit IRC | 23:51 | |
| *** hogepodge has quit IRC | 23:51 | |
| *** madhuvishy has quit IRC | 23:51 | |
| *** dirk has quit IRC | 23:51 | |
| *** ildikov has quit IRC | 23:51 | |
| *** dtantsur|afk has quit IRC | 23:51 | |
| *** tommylikehu has quit IRC | 23:51 | |
| *** geekinutah has quit IRC | 23:51 | |
| *** verdurin has quit IRC | 23:51 | |
| *** sulo has quit IRC | 23:51 | |
| *** evgenyl has quit IRC | 23:51 | |
| *** asilenkov has quit IRC | 23:51 | |
| *** StevenK has quit IRC | 23:51 | |
| *** kencjohnston_ has quit IRC | 23:51 | |
| *** gaelL_ has quit IRC | 23:51 | |
| *** andrewbogott has quit IRC | 23:51 | |
| *** rguillebert has quit IRC | 23:51 | |
| *** ddurst has quit IRC | 23:51 | |
| *** zul has quit IRC | 23:51 | |
| *** sweston has quit IRC | 23:51 | |
| *** petems has quit IRC | 23:51 | |
| *** spzala_ has quit IRC | 23:51 | |
| *** chlong_ has quit IRC | 23:51 | |
| *** claudiub has quit IRC | 23:51 | |
| *** hongbin has quit IRC | 23:51 | |
| *** mudpuppy has quit IRC | 23:51 | |
| *** atarakt has quit IRC | 23:51 | |
| *** kzaitsev1pi has quit IRC | 23:51 | |
| *** abelur has quit IRC | 23:51 | |
| *** pahuang has quit IRC | 23:51 | |
| *** ianw has quit IRC | 23:51 | |
| *** liusheng has quit IRC | 23:51 | |
| *** mnaser has quit IRC | 23:51 | |
| *** fabo has quit IRC | 23:51 | |
| *** weshay has quit IRC | 23:51 | |
| *** vponomaryov has quit IRC | 23:51 | |
| *** med_ has quit IRC | 23:51 | |
| *** alex_xu has quit IRC | 23:51 | |
| *** raginbajin has quit IRC | 23:51 | |
| *** larsks has quit IRC | 23:51 | |
| *** migi has quit IRC | 23:51 | |
| *** _ari_ has quit IRC | 23:51 | |
| *** trown|outtypewww has quit IRC | 23:51 | |
| *** yamamoto has quit IRC | 23:51 | |
| *** ilpianista_ has quit IRC | 23:51 | |
| *** jcoufal has quit IRC | 23:51 | |
| *** sflanigan has quit IRC | 23:51 | |
| *** lathiat has quit IRC | 23:51 | |
| *** GregHouse has quit IRC | 23:51 | |
| *** betherly has quit IRC | 23:51 | |
| *** simonmcc has quit IRC | 23:51 | |
| *** berendt has quit IRC | 23:51 | |
| *** TheJulia has quit IRC | 23:51 | |
| *** Qiming has quit IRC | 23:51 | |
| *** serverascode has quit IRC | 23:51 | |
| *** arif-ali has quit IRC | 23:51 | |
| *** oomichi has quit IRC | 23:51 | |
| *** robcresswell has quit IRC | 23:51 | |
| *** srwilkers_ has quit IRC | 23:51 | |
| *** mugsie has quit IRC | 23:51 | |
| *** zxiiro has quit IRC | 23:51 | |
| *** scottda has quit IRC | 23:51 | |
| *** coreycb has quit IRC | 23:51 | |
| *** patrickeast has quit IRC | 23:51 | |
| *** ameade has quit IRC | 23:51 | |
| *** samueldmq has quit IRC | 23:51 | |
| *** clayg has quit IRC | 23:51 | |
| *** mgkwill has quit IRC | 23:51 | |
| *** xgerman_ has quit IRC | 23:51 | |
| *** fenar has quit IRC | 23:51 | |
| *** zaro has quit IRC | 23:51 | |
| *** cargonza has quit IRC | 23:51 | |
| *** srwilkers has quit IRC | 23:51 | |
| *** vkmc has quit IRC | 23:51 | |
| *** hrybacki has quit IRC | 23:51 | |
| *** v1k0d3n has quit IRC | 23:51 | |
| *** ilpianista has quit IRC | 23:51 | |
| *** MasterofJOKers has quit IRC | 23:51 | |
| *** mgagne has quit IRC | 23:51 | |
| *** kwazar has quit IRC | 23:51 | |
| *** gus has quit IRC | 23:51 | |
| *** fungi has quit IRC | 23:51 | |
| *** leifmadsen has quit IRC | 23:51 | |
| *** tomhambleton_ has quit IRC | 23:51 | |
| *** davidlenwell has quit IRC | 23:51 | |
| *** johnsom has quit IRC | 23:51 | |
| *** vdrok has quit IRC | 23:51 | |
| *** fkautz has quit IRC | 23:51 | |
| *** kklimonda has quit IRC | 23:51 | |
| *** NobodyCam has quit IRC | 23:51 | |
| *** zhongjun has quit IRC | 23:51 | |
| *** armax has quit IRC | 23:51 | |
| *** rwsu has quit IRC | 23:51 | |
| *** spiffxp has quit IRC | 23:51 | |
| *** efried has quit IRC | 23:51 | |
| *** Douhet has quit IRC | 23:51 | |
| *** rockstar has quit IRC | 23:51 | |
| *** portdirect has quit IRC | 23:51 | |
| *** ssbarnea has quit IRC | 23:51 | |
| *** inc0 has quit IRC | 23:51 | |
| *** DinaBelova has quit IRC | 23:51 | |
| *** otherwiseguy has quit IRC | 23:51 | |
| *** russellb has quit IRC | 23:51 | |
| *** SergeyLukjanov has quit IRC | 23:51 | |
| *** filler has quit IRC | 23:51 | |
| *** rakhmerov has quit IRC | 23:51 | |
| *** Odd_Bloke has quit IRC | 23:51 | |
| *** ruhe has quit IRC | 23:51 | |
| *** rodrigods has quit IRC | 23:51 | |
| *** isviridoz has quit IRC | 23:51 | |
| *** sdake has quit IRC | 23:51 | |
| *** Nakato has quit IRC | 23:51 | |
| *** bstinson has quit IRC | 23:51 | |
| *** markmcclain has quit IRC | 23:51 | |
| *** danpawlik has quit IRC | 23:51 | |
| *** mrunge has quit IRC | 23:51 | |
| *** zeus has quit IRC | 23:51 | |
| *** clarkb has quit IRC | 23:51 | |
| *** anthonyper has quit IRC | 23:51 | |
| *** zaneb has quit IRC | 23:51 | |
| *** andreaf has quit IRC | 23:51 | |
| *** vaidy has quit IRC | 23:51 | |
| *** knikolla has quit IRC | 23:51 | |
| *** puiterwijk has quit IRC | 23:51 | |
| *** pots has quit IRC | 23:51 | |
| *** mrhillsman has quit IRC | 23:51 | |
| *** dteselkin has quit IRC | 23:51 | |
| *** katyafervent_awa has quit IRC | 23:51 | |
| *** funzo has quit IRC | 23:51 | |
| *** jaosorior has quit IRC | 23:51 | |
| *** abelur_ has quit IRC | 23:51 | |
| *** jogo has quit IRC | 23:51 | |
| *** cinerama` has quit IRC | 23:51 | |
| *** ianychoi has quit IRC | 23:51 | |
| *** pabelanger has quit IRC | 23:51 | |
| *** gtmanfred has quit IRC | 23:51 | |
| *** mtreinish has quit IRC | 23:51 | |
| *** jesusaur has quit IRC | 23:51 | |
| *** jrich has quit IRC | 23:51 | |
| *** michaelxin has quit IRC | 23:51 | |
| *** bauzas_afk has quit IRC | 23:51 | |
| *** rcarrillocruz has quit IRC | 23:51 | |
| *** hemanthm has quit IRC | 23:51 | |
| *** ajafo has quit IRC | 23:51 | |
| *** eeiden has quit IRC | 23:51 | |
| *** patriciadomin has quit IRC | 23:51 | |
| *** beagles has quit IRC | 23:51 | |
| *** sc` has quit IRC | 23:51 | |
| *** charz has quit IRC | 23:51 | |
| *** yuval has quit IRC | 23:51 | |
| *** szaher has quit IRC | 23:51 | |
| *** loquacities has quit IRC | 23:51 | |
| *** zerick has quit IRC | 23:51 | |
| *** smcginnis has quit IRC | 23:51 | |
| *** Vivek has quit IRC | 23:51 | |
| *** honza has quit IRC | 23:51 | |
| *** SpamapS has quit IRC | 23:51 | |
| *** Anticimex has quit IRC | 23:51 | |
| *** andreykurilin has quit IRC | 23:51 | |
| *** njohnston has quit IRC | 23:51 | |
| *** Guest6666 has quit IRC | 23:51 | |
| *** shu-mutou-AWAY has quit IRC | 23:51 | |
| *** ChanServ has quit IRC | 23:51 | |
| *** jrist has quit IRC | 23:51 | |
| *** lihi has quit IRC | 23:51 | |
| *** mwhahaha has quit IRC | 23:51 | |
| *** logan- has quit IRC | 23:51 | |
| *** gaurangt has quit IRC | 23:51 | |
| *** Kevin_Zheng has quit IRC | 23:51 | |
| *** jbernard has quit IRC | 23:51 | |
| *** tdasilva has quit IRC | 23:51 | |
| *** mancdaz has quit IRC | 23:51 | |
| *** tbarron has quit IRC | 23:51 | |
| *** cmurphy has quit IRC | 23:51 | |
| *** rfolco has quit IRC | 23:51 | |
| *** melwitt has quit IRC | 23:51 | |
| *** zhenguo has quit IRC | 23:51 | |
| *** mrodden has quit IRC | 23:51 | |
| *** masayukig has quit IRC | 23:51 | |
| *** Zara has quit IRC | 23:51 | |
| *** petrovich has quit IRC | 23:51 | |
| *** gothicmindfood has quit IRC | 23:51 | |
| *** nhandler has quit IRC | 23:51 | |
| *** electrical has quit IRC | 23:51 | |
| *** rajinir has quit IRC | 23:51 | |
| *** jpmaxman has quit IRC | 23:51 | |
| *** Ng has quit IRC | 23:51 | |
| *** watersoul has quit IRC | 23:51 | |
| *** mpjetta has quit IRC | 23:51 | |
| *** aviau has quit IRC | 23:51 | |
| *** rm_work has quit IRC | 23:51 | |
| *** onovy has quit IRC | 23:51 | |
| *** bmjen has quit IRC | 23:51 | |
| *** puiterwijk|cld has quit IRC | 23:51 | |
| *** ekhugen has quit IRC | 23:51 | |
| *** quite has quit IRC | 23:51 | |
| *** luzC has quit IRC | 23:51 | |
| *** lennyb has quit IRC | 23:51 | |
| *** stevemar has quit IRC | 23:51 | |
| *** mdrabe has quit IRC | 23:51 | |
| *** stephenfin has quit IRC | 23:51 | |
| *** khappone has quit IRC | 23:51 | |
| *** dtroyer has quit IRC | 23:51 | |
| *** jklare has quit IRC | 23:51 | |
| *** therve has quit IRC | 23:51 | |
| *** jd_ has quit IRC | 23:51 | |
| *** mordred has quit IRC | 23:51 | |
| *** afazekas has quit IRC | 23:51 | |
| *** flwang has quit IRC | 23:51 | |
| *** hemna has quit IRC | 23:51 | |
| *** kmalloc has quit IRC | 23:51 | |
| *** Gorian has quit IRC | 23:51 | |
| *** briancurtin has quit IRC | 23:51 | |
| *** jamespage has quit IRC | 23:51 | |
| *** dham1 has quit IRC | 23:51 | |
| *** auggy has quit IRC | 23:51 | |
| *** kmARC has quit IRC | 23:51 | |
| *** neith has quit IRC | 23:51 | |
| *** jbryce has quit IRC | 23:51 | |
| *** ggherdov- has quit IRC | 23:51 | |
| *** auggy has joined #openstack-infra | 23:55 | |
| *** jbryce has joined #openstack-infra | 23:55 | |
| *** kmalloc has joined #openstack-infra | 23:55 | |
| *** kmARC has joined #openstack-infra | 23:55 | |
| *** briancurtin has joined #openstack-infra | 23:55 | |
| *** DuncanT has joined #openstack-infra | 23:55 | |
| *** jd_ has joined #openstack-infra | 23:55 | |
| *** puiterwijk|cld has joined #openstack-infra | 23:55 | |
| *** mdrabe has joined #openstack-infra | 23:55 | |
| *** quite has joined #openstack-infra | 23:55 | |
| *** flwang has joined #openstack-infra | 23:55 | |
| *** jkilpatr has joined #openstack-infra | 23:55 | |
| *** khappone has joined #openstack-infra | 23:55 | |
| *** jklare has joined #openstack-infra | 23:55 | |
| *** lennyb has joined #openstack-infra | 23:55 | |
| *** mordred has joined #openstack-infra | 23:55 | |
| *** onovy has joined #openstack-infra | 23:55 | |
| *** ekhugen has joined #openstack-infra | 23:55 | |
| *** mpjetta has joined #openstack-infra | 23:55 | |
| *** dtroyer has joined #openstack-infra | 23:55 | |
| *** watersoul has joined #openstack-infra | 23:55 | |
| *** therve has joined #openstack-infra | 23:55 | |
| *** afazekas has joined #openstack-infra | 23:55 | |
| *** bmjen has joined #openstack-infra | 23:55 | |
| *** luzC has joined #openstack-infra | 23:55 | |
| *** Gorian has joined #openstack-infra | 23:55 | |
| *** pbourke has joined #openstack-infra | 23:55 | |
| *** dhill_ has joined #openstack-infra | 23:55 | |
| *** gibi has joined #openstack-infra | 23:55 | |
| *** gouthamr has joined #openstack-infra | 23:55 | |
| *** yamamoto has joined #openstack-infra | 23:55 | |
| *** ilpianista_ has joined #openstack-infra | 23:55 | |
| *** lin_yang has joined #openstack-infra | 23:55 | |
| *** masayukig[m] has joined #openstack-infra | 23:55 | |
| *** Apoorva_ has joined #openstack-infra | 23:55 | |
| *** lewo has joined #openstack-infra | 23:55 | |
| *** d0ugal has joined #openstack-infra | 23:55 | |
| *** armax has joined #openstack-infra | 23:55 | |
| *** jamesden_ has joined #openstack-infra | 23:55 | |
| *** kjackal_ has joined #openstack-infra | 23:55 | |
| *** annegentle has joined #openstack-infra | 23:55 | |
| *** sshnaidm|off has joined #openstack-infra | 23:55 | |
| *** rybridges has joined #openstack-infra | 23:55 | |
| *** yamahata has joined #openstack-infra | 23:55 | |
| *** Julien-zte has joined #openstack-infra | 23:55 | |
| *** Guest80741 has joined #openstack-infra | 23:55 | |
| *** chlong_ has joined #openstack-infra | 23:55 | |
| *** david-lyle has joined #openstack-infra | 23:55 | |
| *** caphrim007 has joined #openstack-infra | 23:55 | |
| *** esberglu has joined #openstack-infra | 23:55 | |
| *** jcoufal has joined #openstack-infra | 23:55 | |
| *** krtaylor has joined #openstack-infra | 23:55 | |
| *** slagle has joined #openstack-infra | 23:55 | |
| *** rlandy has joined #openstack-infra | 23:55 | |
| *** HeOS has joined #openstack-infra | 23:55 | |
| *** flaper87 has joined #openstack-infra | 23:55 | |
| *** AJaeger has joined #openstack-infra | 23:55 | |
| *** rwsu has joined #openstack-infra | 23:55 | |
| *** Dinesh_Bhor has joined #openstack-infra | 23:55 | |
| *** arturb has joined #openstack-infra | 23:55 | |
| *** tnarg has joined #openstack-infra | 23:55 | |
| *** aeng has joined #openstack-infra | 23:55 | |
| *** sflanigan has joined #openstack-infra | 23:55 | |
| *** panda has joined #openstack-infra | 23:55 | |
| *** jrist has joined #openstack-infra | 23:55 | |
| *** Jeffrey4l has joined #openstack-infra | 23:55 | |
| *** lathiat has joined #openstack-infra | 23:55 | |
| *** mandre has joined #openstack-infra | 23:55 | |
| *** akscram has joined #openstack-infra | 23:55 | |
| *** peterlisak has joined #openstack-infra | 23:55 | |
| *** pcrews has joined #openstack-infra | 23:55 | |
| *** makowals has joined #openstack-infra | 23:55 | |
| *** vipul has joined #openstack-infra | 23:55 | |
| *** spiffxp has joined #openstack-infra | 23:55 | |
| *** kuromagi has joined #openstack-infra | 23:55 | |
| *** tinwood has joined #openstack-infra | 23:55 | |
| *** calebb has joined #openstack-infra | 23:55 | |
| *** clayton has joined #openstack-infra | 23:55 | |
| *** mudpuppy has joined #openstack-infra | 23:55 | |
| *** efried_zzz has joined #openstack-infra | 23:55 | |
| *** GregHouse has joined #openstack-infra | 23:55 | |
| *** niska has joined #openstack-infra | 23:55 | |
| *** apetrich has joined #openstack-infra | 23:55 | |
| *** bswartz has joined #openstack-infra | 23:55 | |
| *** lihi has joined #openstack-infra | 23:55 | |
| *** oanson has joined #openstack-infra | 23:55 | |
| *** Hunner has joined #openstack-infra | 23:55 | |
| *** mwhahaha has joined #openstack-infra | 23:55 | |
| *** betherly has joined #openstack-infra | 23:55 | |
| *** simonmcc has joined #openstack-infra | 23:55 | |
| *** fdegir has joined #openstack-infra | 23:55 | |
| *** TheJulia has joined #openstack-infra | 23:55 | |
| *** berendt has joined #openstack-infra | 23:55 | |
| *** rockstar has joined #openstack-infra | 23:55 | |
| *** Douhet has joined #openstack-infra | 23:55 | |
| *** atarakt has joined #openstack-infra | 23:55 | |
| *** benj_ has joined #openstack-infra | 23:55 | |
| *** thorre has joined #openstack-infra | 23:55 | |
| *** hamzy has joined #openstack-infra | 23:55 | |
| *** spligak has joined #openstack-infra | 23:55 | |
| *** kzaitsev1pi has joined #openstack-infra | 23:55 | |
| *** jistr|off has joined #openstack-infra | 23:55 | |
| *** lifeless_ has joined #openstack-infra | 23:55 | |
| *** portdirect has joined #openstack-infra | 23:55 | |
| *** junbo has joined #openstack-infra | 23:55 | |
| *** adriant has joined #openstack-infra | 23:55 | |
| *** Nil_ has joined #openstack-infra | 23:55 | |
| *** mjblack has joined #openstack-infra | 23:55 | |
| *** Diabelko has joined #openstack-infra | 23:55 | |
| *** Adri2000 has joined #openstack-infra | 23:55 | |
| *** ssbarnea has joined #openstack-infra | 23:55 | |
| *** inc0 has joined #openstack-infra | 23:55 | |
| *** mattoliverau has joined #openstack-infra | 23:55 | |
| *** abelur has joined #openstack-infra | 23:55 | |
| *** DinaBelova has joined #openstack-infra | 23:55 | |
| *** filler has joined #openstack-infra | 23:55 | |
| *** otherwiseguy has joined #openstack-infra | 23:55 | |
| *** pahuang has joined #openstack-infra | 23:55 | |
| *** russellb has joined #openstack-infra | 23:55 | |
| *** ianw has joined #openstack-infra | 23:55 | |
| *** niedbalski has joined #openstack-infra | 23:55 | |
| *** SergeyLukjanov has joined #openstack-infra | 23:55 | |
| *** logan- has joined #openstack-infra | 23:55 | |
| *** rbergeron has joined #openstack-infra | 23:55 | |
| *** rakhmerov has joined #openstack-infra | 23:55 | |
| *** larainema has joined #openstack-infra | 23:55 | |
| *** jmorgan1 has joined #openstack-infra | 23:55 | |
| *** ggillies has joined #openstack-infra | 23:55 | |
| *** mpranjic has joined #openstack-infra | 23:55 | |
| *** odyssey4me has joined #openstack-infra | 23:55 | |
| *** Odd_Bloke has joined #openstack-infra | 23:55 | |
| *** ruhe has joined #openstack-infra | 23:55 | |
| *** aarefiev_afk has joined #openstack-infra | 23:55 | |
| *** stevebaker has joined #openstack-infra | 23:55 | |
| *** fbouliane has joined #openstack-infra | 23:55 | |
| *** rodrigods has joined #openstack-infra | 23:55 | |
| *** larsks has joined #openstack-infra | 23:55 | |
| *** beisner has joined #openstack-infra | 23:55 | |
| *** Qiming has joined #openstack-infra | 23:55 | |
| *** raissa has joined #openstack-infra | 23:55 | |
| *** liusheng has joined #openstack-infra | 23:55 | |
| *** mnaser has joined #openstack-infra | 23:55 | |
| *** olaph has joined #openstack-infra | 23:55 | |
| *** serverascode has joined #openstack-infra | 23:55 | |
| *** isviridoz has joined #openstack-infra | 23:55 | |
| *** fabo has joined #openstack-infra | 23:55 | |
| *** weshay has joined #openstack-infra | 23:55 | |
| *** asettle has joined #openstack-infra | 23:55 | |
| *** dutsmoc has joined #openstack-infra | 23:55 | |
| *** sdake has joined #openstack-infra | 23:55 | |
| *** robled has joined #openstack-infra | 23:55 | |
| *** Nakato has joined #openstack-infra | 23:55 | |
| *** cloudnull has joined #openstack-infra | 23:55 | |
| *** mnencia has joined #openstack-infra | 23:55 | |
| *** bstinson has joined #openstack-infra | 23:55 | |
| *** arif-ali has joined #openstack-infra | 23:55 | |
| *** markmcclain has joined #openstack-infra | 23:55 | |
| *** deduped has joined #openstack-infra | 23:55 | |
| *** sambetts|afk has joined #openstack-infra | 23:55 | |
| *** reed has joined #openstack-infra | 23:55 | |
| *** adam_g has joined #openstack-infra | 23:55 | |
| *** sai has joined #openstack-infra | 23:55 | |
| *** isq_ has joined #openstack-infra | 23:55 | |
| *** danpawlik has joined #openstack-infra | 23:55 | |
| *** oomichi has joined #openstack-infra | 23:55 | |
| *** masber has joined #openstack-infra | 23:55 | |
| *** mrunge has joined #openstack-infra | 23:55 | |
| *** fmccrthy has joined #openstack-infra | 23:55 | |
| *** dosaboy has joined #openstack-infra | 23:55 | |
| *** dims has joined #openstack-infra | 23:55 | |
| *** robcresswell has joined #openstack-infra | 23:55 | |
| *** seongsoocho has joined #openstack-infra | 23:55 | |
| *** pfallenop has joined #openstack-infra | 23:55 | |
| *** kaisers2 has joined #openstack-infra | 23:55 | |
| *** thingee has joined #openstack-infra | 23:55 | |
| *** zer0c00l_ has joined #openstack-infra | 23:55 | |
| *** srwilkers_ has joined #openstack-infra | 23:55 | |
| *** zeus has joined #openstack-infra | 23:55 | |
| *** vponomaryov has joined #openstack-infra | 23:55 | |
| *** mugsie has joined #openstack-infra | 23:55 | |
| *** clarkb has joined #openstack-infra | 23:55 | |
| *** eglute has joined #openstack-infra | 23:55 | |
| *** d34dh0r53 has joined #openstack-infra | 23:55 | |
| *** dhellmann has joined #openstack-infra | 23:55 | |
| *** rvba has joined #openstack-infra | 23:55 | |
| *** uberjay has joined #openstack-infra | 23:55 | |
| *** anthonyper has joined #openstack-infra | 23:55 | |
| *** yuanying has joined #openstack-infra | 23:55 | |
| *** bradm has joined #openstack-infra | 23:55 | |
| *** med_ has joined #openstack-infra | 23:55 | |
| *** jmccrory has joined #openstack-infra | 23:55 | |
| *** gorba_ has joined #openstack-infra | 23:55 | |
| *** notmyname has joined #openstack-infra | 23:55 | |
| *** rook has joined #openstack-infra | 23:55 | |
| *** lucas-afk has joined #openstack-infra | 23:55 | |
| *** wfoster has joined #openstack-infra | 23:55 | |
| *** jamielennox has joined #openstack-infra | 23:55 | |
| *** tristanC has joined #openstack-infra | 23:55 | |
| *** yankcrime has joined #openstack-infra | 23:55 | |
| *** rhagarty has joined #openstack-infra | 23:55 | |
| *** timburke has joined #openstack-infra | 23:55 | |
| *** gaurangt has joined #openstack-infra | 23:55 | |
| *** Kevin_Zheng has joined #openstack-infra | 23:55 | |
| *** zaneb has joined #openstack-infra | 23:55 | |
| *** andreaf has joined #openstack-infra | 23:55 | |
| *** mika has joined #openstack-infra | 23:55 | |
| *** cburgess has joined #openstack-infra | 23:55 | |
| *** kong has joined #openstack-infra | 23:55 | |
| *** jbernard has joined #openstack-infra | 23:55 | |
| *** alex_xu has joined #openstack-infra | 23:55 | |
| *** Shrews has joined #openstack-infra | 23:55 | |
| *** greghaynes has joined #openstack-infra | 23:55 | |
| *** vaidy has joined #openstack-infra | 23:55 | |
| *** petevg has joined #openstack-infra | 23:55 | |
| *** raginbajin has joined #openstack-infra | 23:55 | |
| *** antonym has joined #openstack-infra | 23:55 | |
| *** tdasilva has joined #openstack-infra | 23:55 | |
| *** toabctl has joined #openstack-infra | 23:55 | |
| *** mancdaz has joined #openstack-infra | 23:55 | |
| *** freerunner has joined #openstack-infra | 23:55 | |
| *** hogepodge has joined #openstack-infra | 23:55 | |
| *** zxiiro has joined #openstack-infra | 23:55 | |
| *** madhuvishy has joined #openstack-infra | 23:55 | |
| *** scottda has joined #openstack-infra | 23:55 | |
| *** coreycb has joined #openstack-infra | 23:55 | |
| *** dirk has joined #openstack-infra | 23:55 | |
| *** patrickeast has joined #openstack-infra | 23:55 | |
| *** ildikov has joined #openstack-infra | 23:55 | |
| *** mgkwill has joined #openstack-infra | 23:55 | |
| *** samueldmq has joined #openstack-infra | 23:55 | |
| *** ameade has joined #openstack-infra | 23:55 | |
| *** clayg has joined #openstack-infra | 23:55 | |
| *** fenar has joined #openstack-infra | 23:55 | |
| *** xgerman_ has joined #openstack-infra | 23:55 | |
| *** SotK has joined #openstack-infra | 23:55 | |
| *** kklimonda has joined #openstack-infra | 23:55 | |
| *** knikolla has joined #openstack-infra | 23:55 | |
| *** tbarron has joined #openstack-infra | 23:55 | |
| *** cmurphy has joined #openstack-infra | 23:55 | |
| *** dtantsur|afk has joined #openstack-infra | 23:55 | |
| *** zaro has joined #openstack-infra | 23:55 | |
| *** cargonza has joined #openstack-infra | 23:55 | |
| *** srwilkers has joined #openstack-infra | 23:55 | |
| *** tommylikehu has joined #openstack-infra | 23:55 | |
| *** geekinutah has joined #openstack-infra | 23:55 | |
| *** Swimingly has joined #openstack-infra | 23:55 | |
| *** jeblair has joined #openstack-infra | 23:55 | |
| *** verdurin has joined #openstack-infra | 23:55 | |
| *** puiterwijk has joined #openstack-infra | 23:55 | |
| *** dgonzalez has joined #openstack-infra | 23:55 | |
| *** vkmc has joined #openstack-infra | 23:55 | |
| *** rfolco has joined #openstack-infra | 23:55 | |
| *** melwitt has joined #openstack-infra | 23:55 | |
| *** hrybacki has joined #openstack-infra | 23:55 | |
| *** v1k0d3n has joined #openstack-infra | 23:55 | |
| *** zhenguo has joined #openstack-infra | 23:55 | |
| *** tonyb has joined #openstack-infra | 23:55 | |
| *** mhayden has joined #openstack-infra | 23:55 | |
| *** sulo has joined #openstack-infra | 23:55 | |
| *** mrodden has joined #openstack-infra | 23:55 | |
| *** schang has joined #openstack-infra | 23:55 | |
| *** ttx has joined #openstack-infra | 23:55 | |
| *** masayukig has joined #openstack-infra | 23:55 | |
| *** johnthetubaguy has joined #openstack-infra | 23:55 | |
| *** Zara has joined #openstack-infra | 23:55 | |
| *** lamt has joined #openstack-infra | 23:55 | |
| *** ilpianista has joined #openstack-infra | 23:55 | |
| *** devananda has joined #openstack-infra | 23:55 | |
| *** petrovich has joined #openstack-infra | 23:55 | |
| *** eventingmonkey has joined #openstack-infra | 23:55 | |
| *** evgenyl has joined #openstack-infra | 23:55 | |
| *** andymccr has joined #openstack-infra | 23:55 | |
| *** wznoinsk has joined #openstack-infra | 23:55 | |
| *** NikitaKonovalov has joined #openstack-infra | 23:55 | |
| *** laurelm has joined #openstack-infra | 23:55 | |
| *** vgnbkr has joined #openstack-infra | 23:55 | |
| *** peterstac has joined #openstack-infra | 23:55 | |
| *** Keitaro has joined #openstack-infra | 23:55 | |
| *** asilenkov has joined #openstack-infra | 23:55 | |
| *** jlk has joined #openstack-infra | 23:55 | |
| *** timothyb89 has joined #openstack-infra | 23:55 | |
| *** dougshelley66 has joined #openstack-infra | 23:55 | |
| *** ericyoung has joined #openstack-infra | 23:55 | |
| *** bkero has joined #openstack-infra | 23:55 | |
| *** MasterofJOKers has joined #openstack-infra | 23:55 | |
| *** mmedvede has joined #openstack-infra | 23:55 | |
| *** mgagne has joined #openstack-infra | 23:55 | |
| *** StevenK has joined #openstack-infra | 23:55 | |
| *** kwazar has joined #openstack-infra | 23:55 | |
| *** EmilienM has joined #openstack-infra | 23:55 | |
| *** migi has joined #openstack-infra | 23:55 | |
| *** _ari_ has joined #openstack-infra | 23:55 | |
| *** gus has joined #openstack-infra | 23:55 | |
| *** gothicmindfood has joined #openstack-infra | 23:55 | |
| *** zigo has joined #openstack-infra | 23:55 | |
| *** fungi has joined #openstack-infra | 23:55 | |
| *** gaelL_ has joined #openstack-infra | 23:55 | |
| *** nhandler has joined #openstack-infra | 23:55 | |
| *** kencjohnston_ has joined #openstack-infra | 23:55 | |
| *** electrical has joined #openstack-infra | 23:55 | |
| *** rajinir has joined #openstack-infra | 23:55 | |
| *** andrewbogott has joined #openstack-infra | 23:55 | |
| *** rguillebert has joined #openstack-infra | 23:55 | |
| *** leifmadsen has joined #openstack-infra | 23:55 | |
| *** zeroDivisible has joined #openstack-infra | 23:55 | |
| *** davidlenwell has joined #openstack-infra | 23:55 | |
| *** tomhambleton_ has joined #openstack-infra | 23:55 | |
| *** bradjones has joined #openstack-infra | 23:55 | |
| *** ddurst has joined #openstack-infra | 23:55 | |
| *** trown|outtypewww has joined #openstack-infra | 23:55 | |
| *** zul has joined #openstack-infra | 23:55 | |
| *** sweston has joined #openstack-infra | 23:55 | |
| *** johnsom has joined #openstack-infra | 23:55 | |
| *** jpmaxman has joined #openstack-infra | 23:55 | |
| *** vdrok has joined #openstack-infra | 23:55 | |
| *** fkautz has joined #openstack-infra | 23:55 | |
| *** petems has joined #openstack-infra | 23:55 | |
| *** mrhillsman has joined #openstack-infra | 23:55 | |
| *** Ng has joined #openstack-infra | 23:55 | |
| *** pots has joined #openstack-infra | 23:55 | |
| *** zhongjun has joined #openstack-infra | 23:55 | |
| *** NobodyCam has joined #openstack-infra | 23:55 | |
| *** ilpianista_ has quit IRC | 23:55 | |
| *** jamespage has joined #openstack-infra | 23:55 | |
| *** neith has joined #openstack-infra | 23:55 | |
| *** dham1 has joined #openstack-infra | 23:55 | |
| *** rook has quit IRC | 23:55 | |
| *** atarakt has quit IRC | 23:55 | |
| *** masayukig[m] has quit IRC | 23:55 | |
| *** rtjure has quit IRC | 23:55 | |
| *** shu-mutou-AWAY has joined #openstack-infra | 23:56 | |
| *** ChanServ has joined #openstack-infra | 23:56 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 23:56 | |
| *** dmsimard|off has joined #openstack-infra | 23:56 | |
| *** jangutter has joined #openstack-infra | 23:56 | |
| *** openstackgerrit has joined #openstack-infra | 23:56 | |
| *** zzzeek has joined #openstack-infra | 23:56 | |
| *** markmcd has joined #openstack-infra | 23:56 | |
| *** yolanda has joined #openstack-infra | 23:56 | |
| *** wendar has joined #openstack-infra | 23:56 | |
| *** numans has joined #openstack-infra | 23:56 | |
| *** zz_ja has joined #openstack-infra | 23:56 | |
| *** hughsaunders has joined #openstack-infra | 23:56 | |
| *** csatari has joined #openstack-infra | 23:56 | |
| *** arxcruz has joined #openstack-infra | 23:56 | |
| *** edwarnicke has joined #openstack-infra | 23:56 | |
| *** jcook has joined #openstack-infra | 23:56 | |
| *** mrmartin has joined #openstack-infra | 23:56 | |
| *** ggherdov- has joined #openstack-infra | 23:56 | |
| *** trown|outtypewww has quit IRC | 23:56 | |
| *** lastmikoi has joined #openstack-infra | 23:56 | |
| *** dulek has joined #openstack-infra | 23:56 | |
| *** rtjure has joined #openstack-infra | 23:56 | |
| *** dteselkin has joined #openstack-infra | 23:56 | |
| *** katyafervent_awa has joined #openstack-infra | 23:56 | |
| *** funzo has joined #openstack-infra | 23:56 | |
| *** jaosorior has joined #openstack-infra | 23:56 | |
| *** abelur_ has joined #openstack-infra | 23:56 | |
| *** jogo has joined #openstack-infra | 23:56 | |
| *** cinerama` has joined #openstack-infra | 23:56 | |
| *** ianychoi has joined #openstack-infra | 23:56 | |
| *** jrich has joined #openstack-infra | 23:56 | |
| *** pabelanger has joined #openstack-infra | 23:56 | |
| *** gtmanfred has joined #openstack-infra | 23:56 | |
| *** mtreinish has joined #openstack-infra | 23:56 | |
| *** jesusaur has joined #openstack-infra | 23:56 | |
| *** michaelxin has joined #openstack-infra | 23:56 | |
| *** bauzas_afk has joined #openstack-infra | 23:56 | |
| *** rcarrillocruz has joined #openstack-infra | 23:56 | |
| *** hemanthm has joined #openstack-infra | 23:56 | |
| *** ajafo has joined #openstack-infra | 23:56 | |
| *** beagles has joined #openstack-infra | 23:56 | |
| *** patriciadomin has joined #openstack-infra | 23:56 | |
| *** eeiden has joined #openstack-infra | 23:56 | |
| *** sc` has joined #openstack-infra | 23:56 | |
| *** charz has joined #openstack-infra | 23:56 | |
| *** yuval has joined #openstack-infra | 23:56 | |
| *** szaher has joined #openstack-infra | 23:56 | |
| *** loquacities has joined #openstack-infra | 23:56 | |
| *** zerick has joined #openstack-infra | 23:56 | |
| *** smcginnis has joined #openstack-infra | 23:56 | |
| *** Vivek has joined #openstack-infra | 23:56 | |
| *** honza has joined #openstack-infra | 23:56 | |
| *** SpamapS has joined #openstack-infra | 23:56 | |
| *** Anticimex has joined #openstack-infra | 23:56 | |
| *** andreykurilin has joined #openstack-infra | 23:56 | |
| *** njohnston has joined #openstack-infra | 23:56 | |
| *** Guest6666 has joined #openstack-infra | 23:56 | |
| *** shu-mutou-AWAY has quit IRC | 23:56 | |
| *** ChanServ has quit IRC | 23:56 | |
| *** atarakt has joined #openstack-infra | 23:57 | |
| *** lastmikoi has quit IRC | 23:57 | |
| *** dulek has quit IRC | 23:57 | |
| *** chlong_ has quit IRC | 23:57 | |
| *** mudpuppy has quit IRC | 23:57 | |
| *** kzaitsev1pi has quit IRC | 23:57 | |
| *** abelur has quit IRC | 23:57 | |
| *** pahuang has quit IRC | 23:57 | |
| *** ianw has quit IRC | 23:57 | |
| *** liusheng has quit IRC | 23:57 | |
| *** mnaser has quit IRC | 23:57 | |
| *** fabo has quit IRC | 23:57 | |
| *** weshay has quit IRC | 23:57 | |
| *** vponomaryov has quit IRC | 23:57 | |
| *** med_ has quit IRC | 23:57 | |
| *** alex_xu has quit IRC | 23:57 | |
| *** raginbajin has quit IRC | 23:57 | |
| *** larsks has quit IRC | 23:57 | |
| *** migi has quit IRC | 23:57 | |
| *** _ari_ has quit IRC | 23:57 | |
| *** dteselkin has quit IRC | 23:57 | |
| *** katyafervent_awa has quit IRC | 23:57 | |
| *** funzo has quit IRC | 23:57 | |
| *** jaosorior has quit IRC | 23:57 | |
| *** abelur_ has quit IRC | 23:57 | |
| *** jogo has quit IRC | 23:57 | |
| *** cinerama` has quit IRC | 23:57 | |
| *** ianychoi has quit IRC | 23:57 | |
| *** pabelanger has quit IRC | 23:57 | |
| *** gtmanfred has quit IRC | 23:57 | |
| *** mtreinish has quit IRC | 23:57 | |
| *** jesusaur has quit IRC | 23:57 | |
| *** jrich has quit IRC | 23:57 | |
| *** michaelxin has quit IRC | 23:57 | |
| *** bauzas_afk has quit IRC | 23:57 | |
| *** rcarrillocruz has quit IRC | 23:57 | |
| *** hemanthm has quit IRC | 23:57 | |
| *** ajafo has quit IRC | 23:57 | |
| *** patriciadomin has quit IRC | 23:57 | |
| *** beagles has quit IRC | 23:57 | |
| *** sc` has quit IRC | 23:57 | |
| *** eeiden has quit IRC | 23:57 | |
| *** charz has quit IRC | 23:57 | |
| *** yuval has quit IRC | 23:57 | |
| *** szaher has quit IRC | 23:57 | |
| *** loquacities has quit IRC | 23:57 | |
| *** zerick has quit IRC | 23:57 | |
| *** smcginnis has quit IRC | 23:57 | |
| *** Vivek has quit IRC | 23:57 | |
| *** honza has quit IRC | 23:57 | |
| *** SpamapS has quit IRC | 23:57 | |
| *** Anticimex has quit IRC | 23:57 | |
| *** andreykurilin has quit IRC | 23:57 | |
| *** njohnston has quit IRC | 23:57 | |
| *** Guest6666 has quit IRC | 23:57 | |
| *** dmsimard|off has quit IRC | 23:57 | |
| *** openstackgerrit has quit IRC | 23:57 | |
| *** jangutter has quit IRC | 23:57 | |
| *** zzzeek has quit IRC | 23:57 | |
| *** markmcd has quit IRC | 23:57 | |
| *** yolanda has quit IRC | 23:57 | |
| *** wendar has quit IRC | 23:57 | |
| *** numans has quit IRC | 23:57 | |
| *** zz_ja has quit IRC | 23:57 | |
| *** hughsaunders has quit IRC | 23:57 | |
| *** csatari has quit IRC | 23:57 | |
| *** arxcruz has quit IRC | 23:57 | |
| *** edwarnicke has quit IRC | 23:57 | |
| *** jcook has quit IRC | 23:57 | |
| *** mrmartin has quit IRC | 23:57 | |
| *** neith has quit IRC | 23:57 | |
| *** auggy has quit IRC | 23:57 | |
| *** kmalloc has quit IRC | 23:57 | |
| *** briancurtin has quit IRC | 23:57 | |
| *** DuncanT has quit IRC | 23:57 | |
| *** khappone has quit IRC | 23:57 | |
| *** jkilpatr has quit IRC | 23:57 | |
| *** mpjetta has quit IRC | 23:57 | |
| *** onovy has quit IRC | 23:57 | |
| *** bmjen has quit IRC | 23:57 | |
| *** watersoul has quit IRC | 23:57 | |
| *** pbourke has quit IRC | 23:57 | |
| *** dhill_ has quit IRC | 23:57 | |
| *** gibi has quit IRC | 23:57 | |
| *** gouthamr has quit IRC | 23:57 | |
| *** Apoorva_ has quit IRC | 23:57 | |
| *** lewo has quit IRC | 23:57 | |
| *** jamesden_ has quit IRC | 23:57 | |
| *** annegentle has quit IRC | 23:57 | |
| *** yamahata has quit IRC | 23:57 | |
| *** caphrim007 has quit IRC | 23:57 | |
| *** krtaylor has quit IRC | 23:57 | |
| *** rlandy has quit IRC | 23:57 | |
| *** AJaeger has quit IRC | 23:57 | |
| *** Dinesh_Bhor has quit IRC | 23:57 | |
| *** aeng has quit IRC | 23:57 | |
| *** panda has quit IRC | 23:57 | |
| *** Jeffrey4l has quit IRC | 23:57 | |
| *** mandre has quit IRC | 23:57 | |
| *** makowals has quit IRC | 23:57 | |
| *** kuromagi has quit IRC | 23:57 | |
| *** calebb has quit IRC | 23:57 | |
| *** lin_yang has quit IRC | 23:57 | |
| *** kjackal_ has quit IRC | 23:57 | |
| *** Guest80741 has quit IRC | 23:57 | |
| *** arturb has quit IRC | 23:57 | |
| *** spligak has quit IRC | 23:57 | |
| *** adriant has quit IRC | 23:57 | |
| *** mjblack has quit IRC | 23:57 | |
| *** bswartz has quit IRC | 23:57 | |
| *** jmorgan1 has quit IRC | 23:57 | |
| *** odyssey4me has quit IRC | 23:57 | |
| *** stevebaker has quit IRC | 23:57 | |
| *** beisner has quit IRC | 23:57 | |
| *** robled has quit IRC | 23:57 | |
| *** reed has quit IRC | 23:57 | |
| *** thingee has quit IRC | 23:57 | |
| *** gorba_ has quit IRC | 23:57 | |
| *** cburgess has quit IRC | 23:57 | |
| *** kong has quit IRC | 23:57 | |
| *** Shrews has quit IRC | 23:57 | |
| *** antonym has quit IRC | 23:57 | |
| *** atarakt has quit IRC | 23:57 | |
| *** ggherdov- has quit IRC | 23:57 | |
| *** dham1 has quit IRC | 23:57 | |
| *** kmARC has quit IRC | 23:57 | |
| *** puiterwijk|cld has quit IRC | 23:57 | |
| *** jklare has quit IRC | 23:57 | |
| *** ekhugen has quit IRC | 23:57 | |
| *** Gorian has quit IRC | 23:57 | |
| *** dtroyer has quit IRC | 23:57 | |
| *** d0ugal has quit IRC | 23:57 | |
| *** Julien-zte has quit IRC | 23:57 | |
| *** david-lyle has quit IRC | 23:57 | |
| *** akscram has quit IRC | 23:57 | |
| *** peterlisak has quit IRC | 23:57 | |
| *** pcrews has quit IRC | 23:57 | |
| *** tinwood has quit IRC | 23:57 | |
| *** niska has quit IRC | 23:57 | |
| *** fdegir has quit IRC | 23:57 | |
| *** benj_ has quit IRC | 23:57 | |
| *** Adri2000 has quit IRC | 23:57 | |
| *** larainema has quit IRC | 23:57 | |
| *** dutsmoc has quit IRC | 23:57 | |
| *** asettle has quit IRC | 23:57 | |
| *** cloudnull has quit IRC | 23:57 | |
| *** mnencia has quit IRC | 23:57 | |
| *** fmccrthy has quit IRC | 23:57 | |
| *** dims has quit IRC | 23:57 | |
| *** seongsoocho has quit IRC | 23:57 | |
| *** uberjay has quit IRC | 23:57 | |
| *** wfoster has quit IRC | 23:57 | |
| *** jamielennox has quit IRC | 23:57 | |
| *** yankcrime has quit IRC | 23:57 | |
| *** dgonzalez has quit IRC | 23:57 | |
| *** schang has quit IRC | 23:57 | |
| *** ttx has quit IRC | 23:57 | |
| *** lamt has quit IRC | 23:57 | |
| *** andymccr has quit IRC | 23:57 | |
| *** wznoinsk has quit IRC | 23:57 | |
| *** NikitaKonovalov has quit IRC | 23:57 | |
| *** laurelm has quit IRC | 23:57 | |
| *** vgnbkr has quit IRC | 23:57 | |
| *** peterstac has quit IRC | 23:57 | |
| *** dougshelley66 has quit IRC | 23:57 | |
| *** bkero has quit IRC | 23:57 | |
| *** mmedvede has quit IRC | 23:57 | |
| *** sshnaidm|off has quit IRC | 23:57 | |
| *** rybridges has quit IRC | 23:57 | |
| *** HeOS has quit IRC | 23:57 | |
| *** flaper87 has quit IRC | 23:57 | |
| *** tnarg has quit IRC | 23:57 | |
| *** vipul has quit IRC | 23:57 | |
| *** clayton has quit IRC | 23:57 | |
| *** Hunner has quit IRC | 23:57 | |
| *** olaph has quit IRC | 23:57 | |
| *** isq_ has quit IRC | 23:57 | |
| *** masber has quit IRC | 23:57 | |
| *** dosaboy has quit IRC | 23:57 | |
| *** kaisers2 has quit IRC | 23:57 | |
| *** yuanying has quit IRC | 23:57 | |
| *** bradm has quit IRC | 23:57 | |
| *** devananda has quit IRC | 23:57 | |
| *** timothyb89 has quit IRC | 23:57 | |
| *** EmilienM has quit IRC | 23:57 | |
| *** zigo has quit IRC | 23:57 | |
| *** rtjure has quit IRC | 23:57 | |
| *** flwang has quit IRC | 23:57 | |
| *** quite has quit IRC | 23:57 | |
| *** luzC has quit IRC | 23:57 | |
| *** esberglu has quit IRC | 23:57 | |
| *** slagle has quit IRC | 23:57 | |
| *** oanson has quit IRC | 23:57 | |
| *** thorre has quit IRC | 23:57 | |
| *** hamzy has quit IRC | 23:57 | |
| *** Nil_ has quit IRC | 23:57 | |
| *** mpranjic has quit IRC | 23:57 | |
| *** aarefiev_afk has quit IRC | 23:57 | |
| *** raissa has quit IRC | 23:57 | |
| *** deduped has quit IRC | 23:57 | |
| *** sambetts|afk has quit IRC | 23:57 | |
| *** rvba has quit IRC | 23:57 | |
| *** Diabelko has quit IRC | 23:57 | |
| *** tristanC has quit IRC | 23:57 | |
| *** rhagarty has quit IRC | 23:57 | |
| *** timburke has quit IRC | 23:57 | |
| *** mika has quit IRC | 23:57 | |
| *** toabctl has quit IRC | 23:57 | |
| *** freerunner has quit IRC | 23:57 | |
| *** Swimingly has quit IRC | 23:57 | |
| *** jeblair has quit IRC | 23:57 | |
| *** tonyb has quit IRC | 23:57 | |
| *** mhayden has quit IRC | 23:57 | |
| *** johnthetubaguy has quit IRC | 23:57 | |
| *** eventingmonkey has quit IRC | 23:57 | |
| *** Keitaro has quit IRC | 23:57 | |
| *** jlk has quit IRC | 23:57 | |
| *** ericyoung has quit IRC | 23:57 | |
| *** zer0c00l_ has quit IRC | 23:57 | |
| *** SotK has quit IRC | 23:57 | |
| *** zeroDivisible has quit IRC | 23:57 | |
| *** bradjones has quit IRC | 23:57 | |
| *** jamespage has quit IRC | 23:57 | |
| *** jbryce has quit IRC | 23:57 | |
| *** mdrabe has quit IRC | 23:57 | |
| *** mordred has quit IRC | 23:57 | |
| *** lennyb has quit IRC | 23:57 | |
| *** afazekas has quit IRC | 23:57 | |
| *** therve has quit IRC | 23:57 | |
| *** apetrich has quit IRC | 23:57 | |
| *** jistr|off has quit IRC | 23:57 | |
| *** lifeless_ has quit IRC | 23:57 | |
| *** junbo has quit IRC | 23:57 | |
| *** mattoliverau has quit IRC | 23:57 | |
| *** rbergeron has quit IRC | 23:57 | |
| *** niedbalski has quit IRC | 23:57 | |
| *** ggillies has quit IRC | 23:57 | |
| *** fbouliane has quit IRC | 23:57 | |
| *** adam_g has quit IRC | 23:57 | |
| *** pfallenop has quit IRC | 23:57 | |
| *** eglute has quit IRC | 23:57 | |
| *** d34dh0r53 has quit IRC | 23:57 | |
| *** dhellmann has quit IRC | 23:57 | |
| *** jmccrory has quit IRC | 23:57 | |
| *** notmyname has quit IRC | 23:57 | |
| *** lucas-afk has quit IRC | 23:57 | |
| *** sai has quit IRC | 23:57 | |
| *** greghaynes has quit IRC | 23:57 | |
| *** petevg has quit IRC | 23:57 | |
| *** hogepodge has quit IRC | 23:57 | |
| *** madhuvishy has quit IRC | 23:57 | |
| *** dirk has quit IRC | 23:57 | |
| *** ildikov has quit IRC | 23:57 | |
| *** dtantsur|afk has quit IRC | 23:57 | |
| *** tommylikehu has quit IRC | 23:57 | |
| *** geekinutah has quit IRC | 23:57 | |
| *** verdurin has quit IRC | 23:57 | |
| *** sulo has quit IRC | 23:57 | |
| *** evgenyl has quit IRC | 23:57 | |
| *** asilenkov has quit IRC | 23:57 | |
| *** StevenK has quit IRC | 23:57 | |
| *** kencjohnston_ has quit IRC | 23:57 | |
| *** gaelL_ has quit IRC | 23:57 | |
| *** andrewbogott has quit IRC | 23:57 | |
| *** rguillebert has quit IRC | 23:57 | |
| *** ddurst has quit IRC | 23:57 | |
| *** zul has quit IRC | 23:57 | |
| *** sweston has quit IRC | 23:57 | |
| *** petems has quit IRC | 23:57 | |
| *** jd_ has quit IRC | 23:57 | |
| *** yamamoto has quit IRC | 23:57 | |
| *** jcoufal has quit IRC | 23:57 | |
| *** sflanigan has quit IRC | 23:57 | |
| *** lathiat has quit IRC | 23:57 | |
| *** GregHouse has quit IRC | 23:57 | |
| *** betherly has quit IRC | 23:57 | |
| *** simonmcc has quit IRC | 23:57 | |
| *** berendt has quit IRC | 23:57 | |
| *** TheJulia has quit IRC | 23:57 | |
| *** Qiming has quit IRC | 23:57 | |
| *** serverascode has quit IRC | 23:57 | |
| *** arif-ali has quit IRC | 23:57 | |
| *** oomichi has quit IRC | 23:57 | |
| *** robcresswell has quit IRC | 23:57 | |
| *** srwilkers_ has quit IRC | 23:57 | |
| *** mugsie has quit IRC | 23:57 | |
| *** zxiiro has quit IRC | 23:57 | |
| *** scottda has quit IRC | 23:57 | |
| *** coreycb has quit IRC | 23:57 | |
| *** patrickeast has quit IRC | 23:57 | |
| *** ameade has quit IRC | 23:57 | |
| *** samueldmq has quit IRC | 23:57 | |
| *** clayg has quit IRC | 23:57 | |
| *** mgkwill has quit IRC | 23:57 | |
| *** xgerman_ has quit IRC | 23:57 | |
| *** fenar has quit IRC | 23:57 | |
| *** zaro has quit IRC | 23:57 | |
| *** cargonza has quit IRC | 23:57 | |
| *** srwilkers has quit IRC | 23:57 | |
| *** vkmc has quit IRC | 23:57 | |
| *** hrybacki has quit IRC | 23:57 | |
| *** v1k0d3n has quit IRC | 23:57 | |
| *** ilpianista has quit IRC | 23:57 | |
| *** MasterofJOKers has quit IRC | 23:57 | |
| *** mgagne has quit IRC | 23:57 | |
| *** kwazar has quit IRC | 23:57 | |
| *** gus has quit IRC | 23:57 | |
| *** fungi has quit IRC | 23:57 | |
| *** leifmadsen has quit IRC | 23:57 | |
| *** tomhambleton_ has quit IRC | 23:57 | |
| *** davidlenwell has quit IRC | 23:57 | |
| *** johnsom has quit IRC | 23:57 | |
| *** vdrok has quit IRC | 23:57 | |
| *** fkautz has quit IRC | 23:57 | |
| *** kklimonda has quit IRC | 23:57 | |
| *** NobodyCam has quit IRC | 23:57 | |
| *** zhongjun has quit IRC | 23:57 | |
| *** armax has quit IRC | 23:57 | |
| *** rwsu has quit IRC | 23:57 | |
| *** spiffxp has quit IRC | 23:57 | |
| *** efried_zzz has quit IRC | 23:57 | |
| *** Douhet has quit IRC | 23:57 | |
| *** rockstar has quit IRC | 23:57 | |
| *** portdirect has quit IRC | 23:57 | |
| *** ssbarnea has quit IRC | 23:57 | |
| *** inc0 has quit IRC | 23:57 | |
| *** DinaBelova has quit IRC | 23:57 | |
| *** otherwiseguy has quit IRC | 23:57 | |
| *** russellb has quit IRC | 23:57 | |
| *** SergeyLukjanov has quit IRC | 23:57 | |
| *** filler has quit IRC | 23:57 | |
| *** rakhmerov has quit IRC | 23:57 | |
| *** Odd_Bloke has quit IRC | 23:57 | |
| *** ruhe has quit IRC | 23:57 | |
| *** rodrigods has quit IRC | 23:57 | |
| *** isviridoz has quit IRC | 23:57 | |
| *** sdake has quit IRC | 23:57 | |
| *** Nakato has quit IRC | 23:57 | |
| *** bstinson has quit IRC | 23:57 | |
| *** markmcclain has quit IRC | 23:57 | |
| *** danpawlik has quit IRC | 23:57 | |
| *** mrunge has quit IRC | 23:57 | |
| *** zeus has quit IRC | 23:57 | |
| *** clarkb has quit IRC | 23:57 | |
| *** anthonyper has quit IRC | 23:57 | |
| *** zaneb has quit IRC | 23:57 | |
| *** andreaf has quit IRC | 23:57 | |
| *** vaidy has quit IRC | 23:57 | |
| *** knikolla has quit IRC | 23:57 | |
| *** puiterwijk has quit IRC | 23:57 | |
| *** pots has quit IRC | 23:57 | |
| *** mrhillsman has quit IRC | 23:57 | |
| *** jrist has quit IRC | 23:57 | |
| *** lihi has quit IRC | 23:57 | |
| *** mwhahaha has quit IRC | 23:57 | |
| *** logan- has quit IRC | 23:57 | |
| *** gaurangt has quit IRC | 23:57 | |
| *** Kevin_Zheng has quit IRC | 23:57 | |
| *** jbernard has quit IRC | 23:57 | |
| *** tdasilva has quit IRC | 23:57 | |
| *** mancdaz has quit IRC | 23:57 | |
| *** tbarron has quit IRC | 23:57 | |
| *** cmurphy has quit IRC | 23:57 | |
| *** rfolco has quit IRC | 23:57 | |
| *** melwitt has quit IRC | 23:57 | |
| *** zhenguo has quit IRC | 23:57 | |
| *** mrodden has quit IRC | 23:57 | |
| *** masayukig has quit IRC | 23:57 | |
| *** Zara has quit IRC | 23:57 | |
| *** petrovich has quit IRC | 23:57 | |
| *** gothicmindfood has quit IRC | 23:57 | |
| *** nhandler has quit IRC | 23:57 | |
| *** electrical has quit IRC | 23:57 | |
| *** rajinir has quit IRC | 23:57 | |
| *** jpmaxman has quit IRC | 23:57 | |
| *** Ng has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!