| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: move queries.yaml into a queries subdir https://review.openstack.org/59305 | 00:04 |
|---|---|---|
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: Fix E122,E126,E128 items in codebase https://review.openstack.org/59272 | 00:04 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: method extraction for readability https://review.openstack.org/59263 | 00:04 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: time to grow up and fit in 80 columns https://review.openstack.org/59262 | 00:04 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: wrapping the README.rst file to 80 cols https://review.openstack.org/59261 | 00:04 |
| *** senk has joined #openstack-infra | 00:12 | |
| *** dstanek_afk has joined #openstack-infra | 00:14 | |
| *** sarob has joined #openstack-infra | 00:15 | |
| *** dkliban has joined #openstack-infra | 00:17 | |
| *** rongze has joined #openstack-infra | 00:18 | |
| *** sarob has quit IRC | 00:18 | |
| *** sarob has joined #openstack-infra | 00:19 | |
| *** dstanek_afk has quit IRC | 00:19 | |
| *** matsuhashi has joined #openstack-infra | 00:19 | |
| *** sarob has quit IRC | 00:23 | |
| *** rongze has quit IRC | 00:23 | |
| *** matsuhashi has quit IRC | 00:30 | |
| *** matsuhashi has joined #openstack-infra | 00:30 | |
| *** alexpilotti has quit IRC | 00:30 | |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Savanna client is now using separated LP project https://review.openstack.org/57752 | 00:35 |
| *** matsuhashi has quit IRC | 00:35 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove explicit depends on transitive pip deps https://review.openstack.org/51402 | 00:37 |
| *** jcooley_ has quit IRC | 00:42 | |
| *** alexpilotti has joined #openstack-infra | 00:42 | |
| lifeless | what github repos hold official program code | 00:45 |
| lifeless | openstack-infra/ | 00:45 |
| lifeless | openstack/ | 00:45 |
| lifeless | devstack/ | 00:45 |
| lifeless | anything else ? | 00:45 |
| lifeless | s/github/blah/ | 00:45 |
| *** zul has joined #openstack-infra | 00:46 | |
| sdague | openstack, openstack-dev, openstack-infra | 00:47 |
| sdague | devstack is under openstack-dev | 00:47 |
| lifeless | ah, thanks | 00:47 |
| *** CaptTofu has quit IRC | 00:48 | |
| cyeoh | sdague or any other nova core around - would you mind looking at https://review.openstack.org/#/c/59142/ - its fixes a bug which has been causing failures in the gate | 00:49 |
| *** CaptTofu has joined #openstack-infra | 00:49 | |
| *** sarob has joined #openstack-infra | 00:49 | |
| sdague | cyeoh: sure, lgtm | 00:51 |
| cyeoh | sdague: thx! | 00:51 |
| *** jcooley_ has joined #openstack-infra | 00:52 | |
| *** jcooley_ has quit IRC | 00:58 | |
| *** jcooley_ has joined #openstack-infra | 00:59 | |
| *** jcooley_ has quit IRC | 01:04 | |
| *** matsuhashi has joined #openstack-infra | 01:05 | |
| *** dstanek_afk has joined #openstack-infra | 01:07 | |
| *** adalbas has quit IRC | 01:09 | |
| *** odyssey4me has joined #openstack-infra | 01:15 | |
| *** sarob has quit IRC | 01:21 | |
| *** alexpilotti has quit IRC | 01:22 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: Fix E122,E126,E128 items in codebase https://review.openstack.org/59272 | 01:25 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: move queries.yaml into a queries subdir https://review.openstack.org/59252 | 01:25 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: method extraction for readability https://review.openstack.org/59263 | 01:25 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: time to grow up and fit in 80 columns https://review.openstack.org/59262 | 01:25 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: wrapping the README.rst file to 80 cols https://review.openstack.org/59261 | 01:25 |
| *** marun has joined #openstack-infra | 01:27 | |
| *** lianglin has joined #openstack-infra | 01:32 | |
| *** nosnos has joined #openstack-infra | 01:33 | |
| *** dims has joined #openstack-infra | 01:41 | |
| sdague | mordred: who can I stab for the fact that the foundation membership database is broken, and won't let me update email addresses until I add an affiliation, which it then fails to do anything when I click save? | 01:43 |
| sdague | also note: adding an affiliation is broken | 01:44 |
| *** yaguang has joined #openstack-infra | 01:53 | |
| *** dims has quit IRC | 01:54 | |
| *** bingbu has joined #openstack-infra | 01:57 | |
| *** ljjjustin has joined #openstack-infra | 01:59 | |
| *** rongze has joined #openstack-infra | 02:01 | |
| *** rcleere has joined #openstack-infra | 02:01 | |
| *** mgagne has quit IRC | 02:12 | |
| *** dcramer_ has joined #openstack-infra | 02:12 | |
| *** mgagne1 has joined #openstack-infra | 02:12 | |
| *** mgagne1 has quit IRC | 02:12 | |
| *** mgagne1 has joined #openstack-infra | 02:12 | |
| *** sarob has joined #openstack-infra | 02:19 | |
| *** jishaomei has joined #openstack-infra | 02:21 | |
| *** bingbu has quit IRC | 02:23 | |
| *** sarob has quit IRC | 02:24 | |
| *** CaptTofu has quit IRC | 02:29 | |
| *** CaptTofu has joined #openstack-infra | 02:29 | |
| *** bingbu has joined #openstack-infra | 02:35 | |
| *** guohliu has joined #openstack-infra | 02:36 | |
| *** dkliban has quit IRC | 02:43 | |
| *** slong has quit IRC | 02:46 | |
| *** slong has joined #openstack-infra | 02:46 | |
| *** marun has quit IRC | 02:56 | |
| *** nati_ueno has joined #openstack-infra | 02:57 | |
| *** matsuhashi has quit IRC | 02:59 | |
| clarkb | sdague: fungi and jeblair are your closest people to stab on irc | 03:01 |
| *** sarob has joined #openstack-infra | 03:02 | |
| *** matsuhashi has joined #openstack-infra | 03:02 | |
| *** odyssey4me has quit IRC | 03:03 | |
| *** jcooley_ has joined #openstack-infra | 03:10 | |
| *** dstanek_afk has quit IRC | 03:25 | |
| *** mgagne1 has quit IRC | 03:26 | |
| *** CaptTofu has quit IRC | 03:26 | |
| mordred | sdague: everyone | 03:32 |
| mordred | sdague: I've been trying for a year to get that moved under infra | 03:33 |
| *** bingbu has quit IRC | 03:43 | |
| *** dstanek_afk has joined #openstack-infra | 03:51 | |
| *** nicedice has quit IRC | 03:53 | |
| *** bingbu has joined #openstack-infra | 03:55 | |
| *** dstanek_afk has quit IRC | 03:56 | |
| *** jcooley_ has quit IRC | 03:59 | |
| *** matsuhashi has quit IRC | 04:02 | |
| *** matsuhas_ has joined #openstack-infra | 04:05 | |
| *** markmcclain1 has joined #openstack-infra | 04:12 | |
| *** markmcclain has quit IRC | 04:12 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:14 | |
| *** sarob has quit IRC | 04:16 | |
| *** sarob has joined #openstack-infra | 04:16 | |
| *** sarob has quit IRC | 04:21 | |
| *** chandankumar has quit IRC | 04:28 | |
| *** chandankumar has joined #openstack-infra | 04:34 | |
| *** lianglin has quit IRC | 04:40 | |
| *** sarob has joined #openstack-infra | 04:47 | |
| *** dstanek_afk has joined #openstack-infra | 04:54 | |
| *** DennyZhang has joined #openstack-infra | 04:55 | |
| *** boris-42 has joined #openstack-infra | 04:55 | |
| *** rongze has quit IRC | 04:57 | |
| *** slong has quit IRC | 04:58 | |
| *** dcramer_ has quit IRC | 04:59 | |
| *** dstanek_afk has quit IRC | 04:59 | |
| *** rcleere has quit IRC | 04:59 | |
| *** jorisroovers has joined #openstack-infra | 05:02 | |
| *** jroovers has joined #openstack-infra | 05:06 | |
| *** bingbu has quit IRC | 05:06 | |
| *** jorisroovers has quit IRC | 05:09 | |
| *** sarob has quit IRC | 05:13 | |
| *** sarob has joined #openstack-infra | 05:13 | |
| *** sshturm has quit IRC | 05:17 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 05:19 | |
| *** _SergeyLukjanov has quit IRC | 05:19 | |
| *** bingbu has joined #openstack-infra | 05:20 | |
| *** jcooley_ has joined #openstack-infra | 05:26 | |
| *** talluri has joined #openstack-infra | 05:26 | |
| *** rongze has joined #openstack-infra | 05:27 | |
| *** jcooley_ has quit IRC | 05:34 | |
| *** jcooley_ has joined #openstack-infra | 05:34 | |
| *** nosnos_ has joined #openstack-infra | 05:35 | |
| *** bingbu has quit IRC | 05:35 | |
| *** rongze has quit IRC | 05:36 | |
| *** nosnos has quit IRC | 05:37 | |
| *** nosnos_ has quit IRC | 05:37 | |
| *** nosnos has joined #openstack-infra | 05:38 | |
| *** slong has joined #openstack-infra | 05:38 | |
| *** bingbu has joined #openstack-infra | 05:47 | |
| *** jcooley_ has quit IRC | 05:54 | |
| *** jcooley_ has joined #openstack-infra | 05:55 | |
| *** matsuhas_ has quit IRC | 05:55 | |
| *** matsuhashi has joined #openstack-infra | 05:56 | |
| *** rongze has joined #openstack-infra | 06:02 | |
| *** paul-- has quit IRC | 06:10 | |
| *** rongze has quit IRC | 06:11 | |
| *** rongze has joined #openstack-infra | 06:11 | |
| *** sarob has quit IRC | 06:17 | |
| *** sarob has joined #openstack-infra | 06:18 | |
| *** SergeyLukjanov has joined #openstack-infra | 06:20 | |
| *** sarob has quit IRC | 06:22 | |
| *** chandankumar is now known as ciypro | 06:35 | |
| *** bingbu has quit IRC | 06:37 | |
| *** liang has joined #openstack-infra | 06:42 | |
| *** nati_ueno has quit IRC | 06:42 | |
| *** DinaBelova has joined #openstack-infra | 06:44 | |
| liang | Hi, I have a patch merged for pbr last week, https://review.openstack.org/#/c/57892/. | 06:44 |
| liang | Anyone has an idea when this will be synced into the build environment(gate keeping)? | 06:44 |
| pabelanger | liang, I would imagine once they tag a new pbr release | 06:46 |
| *** sarob has joined #openstack-infra | 06:48 | |
| *** jcooley_ has quit IRC | 06:48 | |
| *** jcooley_ has joined #openstack-infra | 06:49 | |
| mordred | we're getting close to that | 06:49 |
| mordred | I want a few more patches to land -then we'll cut a new release | 06:50 |
| mordred | liang: ^^ | 06:50 |
| liang | yup | 06:50 |
| liang | pabelanger, mordred, Thanks for the info! | 06:50 |
| liang | mordred, Will there be an email announce for a new release (the next release)? | 06:51 |
| *** jcooley_ has quit IRC | 06:52 | |
| *** jcooley_ has joined #openstack-infra | 06:52 | |
| *** sarob has quit IRC | 06:54 | |
| *** bingbu has joined #openstack-infra | 06:54 | |
| *** matsuhashi has quit IRC | 06:56 | |
| *** matsuhashi has joined #openstack-infra | 06:56 | |
| *** matsuhashi has quit IRC | 06:57 | |
| *** matsuhashi has joined #openstack-infra | 06:57 | |
| *** nosnos_ has joined #openstack-infra | 06:57 | |
| *** nosnos has quit IRC | 06:57 | |
| *** marun has joined #openstack-infra | 06:59 | |
| mordred | liang: hrm. I should probably do that | 07:01 |
| mordred | liang: I normally just break everyone | 07:01 |
| *** yolanda has quit IRC | 07:02 | |
| liang | mordred, I see. Thanks! | 07:03 |
| mordred | liang: ;) yes - we'll do an announce for sure | 07:03 |
| mordred | zaro: WOOT | 07:06 |
| mordred | just saw the bug update | 07:07 |
| mordred | zaro: that's the best thing EVAR | 07:07 |
| *** ljjjustin has quit IRC | 07:09 | |
| *** lianglin has joined #openstack-infra | 07:11 | |
| jishaomei | Hi,all.I met a question about gearman when I built our ci enviromen. log info ERROR zuul.Gearman: Job <gear.Job 0x7f919803ca90 handle: None name: build:gate-tempest-python27 unique: ab9175d0c2ba401f93ad76e9de134d97> is not registered with Gearman.Would you please do me a favor about how to fix this issue, thanks. | 07:11 |
| *** ljjjustin has joined #openstack-infra | 07:11 | |
| *** jcooley_ has quit IRC | 07:12 | |
| *** jcooley_ has joined #openstack-infra | 07:12 | |
| *** jcooley_ has quit IRC | 07:17 | |
| *** sarob has joined #openstack-infra | 07:18 | |
| *** jcooley_ has joined #openstack-infra | 07:18 | |
| *** nati_ueno has joined #openstack-infra | 07:20 | |
| *** nati_ueno has quit IRC | 07:21 | |
| lianglin | hi guys, anybody can answer shaomei's question above? we need a way to debug why Gearman can't invoke jenkins job | 07:21 |
| lianglin | it would be great if somebody can show step by step debug here :) | 07:23 |
| *** jcooley_ has quit IRC | 07:23 | |
| *** jcooley_ has joined #openstack-infra | 07:23 | |
| lianglin | @fifieldt, @fungi~ can you help? | 07:24 |
| fifieldt | hi | 07:24 |
| fifieldt | unfortunately I'm not expert enough for this question :( | 07:24 |
| lianglin | ha~ anybody you know may be the expert? | 07:25 |
| *** DennyZha` has joined #openstack-infra | 07:25 | |
| *** DennyZhang has quit IRC | 07:25 | |
| fifieldt | I suspect the fungi, mordred, jeblair might know - but they are likely asleep at the moment | 07:25 |
| *** zenoway has joined #openstack-infra | 07:25 | |
| lianglin | got it~ go through the code at first before the experts respons | 07:26 |
| *** zigo has quit IRC | 07:27 | |
| *** zigo has joined #openstack-infra | 07:27 | |
| *** senk has quit IRC | 07:27 | |
| *** bingbu has quit IRC | 07:27 | |
| *** pblaho has joined #openstack-infra | 07:29 | |
| *** jcooley_ has quit IRC | 07:32 | |
| *** jcooley_ has joined #openstack-infra | 07:32 | |
| *** DennyZha` has quit IRC | 07:33 | |
| *** jcooley_ has quit IRC | 07:34 | |
| *** jcooley_ has joined #openstack-infra | 07:36 | |
| *** jcoufal has joined #openstack-infra | 07:40 | |
| *** bingbu has joined #openstack-infra | 07:42 | |
| *** yolanda has joined #openstack-infra | 07:42 | |
| *** avishayb has joined #openstack-infra | 07:43 | |
| *** jcooley_ has quit IRC | 07:43 | |
| *** jcooley_ has joined #openstack-infra | 07:44 | |
| *** sarob has quit IRC | 07:50 | |
| *** jcooley_ has quit IRC | 07:50 | |
| *** jcooley_ has joined #openstack-infra | 07:51 | |
| *** zigo has quit IRC | 07:55 | |
| *** zigo has joined #openstack-infra | 07:55 | |
| *** dstanek_afk has joined #openstack-infra | 07:57 | |
| *** jcooley_ has quit IRC | 07:58 | |
| *** jcooley_ has joined #openstack-infra | 08:00 | |
| *** dstanek_afk has quit IRC | 08:01 | |
| *** nati_ueno has joined #openstack-infra | 08:02 | |
| *** osanchez has joined #openstack-infra | 08:06 | |
| *** DinaBelova has quit IRC | 08:08 | |
| *** sergmelikyan has quit IRC | 08:08 | |
| *** jcooley_ has quit IRC | 08:08 | |
| *** SergeyLukjanov has quit IRC | 08:08 | |
| *** jcooley_ has joined #openstack-infra | 08:08 | |
| *** matsuhashi has quit IRC | 08:10 | |
| *** flaper87|afk is now known as flaper87 | 08:15 | |
| *** SergeyLukjanov has joined #openstack-infra | 08:15 | |
| *** matsuhashi has joined #openstack-infra | 08:16 | |
| *** dizquierdo has joined #openstack-infra | 08:17 | |
| *** sarob has joined #openstack-infra | 08:18 | |
| *** hashar has joined #openstack-infra | 08:19 | |
| *** jcooley_ has quit IRC | 08:21 | |
| *** zenoway has quit IRC | 08:22 | |
| *** osanchez has quit IRC | 08:22 | |
| *** sarob has quit IRC | 08:23 | |
| *** nati_ueno has quit IRC | 08:23 | |
| *** osanchez has joined #openstack-infra | 08:23 | |
| *** DinaBelova has joined #openstack-infra | 08:24 | |
| *** nosnos_ has quit IRC | 08:27 | |
| *** nosnos has joined #openstack-infra | 08:28 | |
| *** dizquierdo has quit IRC | 08:39 | |
| *** avishayb has quit IRC | 08:40 | |
| *** ilyashakhat_ has joined #openstack-infra | 08:42 | |
| *** rongze has quit IRC | 08:44 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 08:48 | |
| *** _SergeyLukjanov has quit IRC | 08:49 | |
| *** SergeyLukjanov has joined #openstack-infra | 08:53 | |
| *** che-arne has quit IRC | 08:55 | |
| *** nati_ueno has joined #openstack-infra | 08:55 | |
| *** yassine has joined #openstack-infra | 09:00 | |
| *** avishayb has joined #openstack-infra | 09:01 | |
| *** hashar has quit IRC | 09:03 | |
| *** markmc has joined #openstack-infra | 09:03 | |
| *** zigo has quit IRC | 09:06 | |
| *** zigo has joined #openstack-infra | 09:06 | |
| *** jpich has joined #openstack-infra | 09:09 | |
| *** matsuhashi has quit IRC | 09:14 | |
| *** bingbu has quit IRC | 09:16 | |
| *** sarob has joined #openstack-infra | 09:18 | |
| *** DinaBelova has quit IRC | 09:19 | |
| *** nati_ueno has quit IRC | 09:19 | |
| *** SergeyLukjanov has quit IRC | 09:20 | |
| *** fbo_away is now known as fbo | 09:20 | |
| *** derekh has joined #openstack-infra | 09:22 | |
| *** afazekas has joined #openstack-infra | 09:22 | |
| *** sarob has quit IRC | 09:23 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:23 | |
| *** hashar has joined #openstack-infra | 09:25 | |
| *** SergeyLukjanov has quit IRC | 09:29 | |
| *** matsuhashi has joined #openstack-infra | 09:31 | |
| Adri2000 | hi | 09:41 |
| Adri2000 | I recently changed my affiliation (updated my profile at openstack.org) but I noticed that my first merged contribution appears with my previous affiliation at http://activity.openstack.org/data | 09:43 |
| Adri2000 | how can I fix this? | 09:44 |
| *** DinaBelova has joined #openstack-infra | 09:51 | |
| *** dizquierdo has joined #openstack-infra | 09:51 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-dev/hacking: Add a check for file with only comments https://review.openstack.org/59090 | 09:52 |
| *** mancdaz has quit IRC | 09:53 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:55 | |
| *** mkulkin has joined #openstack-infra | 09:58 | |
| *** dstanek_afk has joined #openstack-infra | 09:58 | |
| *** dstanek_afk has quit IRC | 10:03 | |
| *** jroovers has quit IRC | 10:05 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Enable #openstack-climate IRC channel logging https://review.openstack.org/57675 | 10:07 |
| *** sarob has joined #openstack-infra | 10:18 | |
| *** lianglin has quit IRC | 10:20 | |
| *** avishayb has quit IRC | 10:20 | |
| katyafervent | Hello all | 10:28 |
| *** Ryan_Lane has quit IRC | 10:28 | |
| *** jorisroovers has joined #openstack-infra | 10:30 | |
| katyafervent | Could somebody help me please with branch creation? There is no opportunity to create new branch (from review.openstack -> Admin) any more. What can be the problem? | 10:30 |
| *** markmcclain1 has quit IRC | 10:33 | |
| *** andreaf has quit IRC | 10:35 | |
| *** andreaf has joined #openstack-infra | 10:35 | |
| *** jorisroovers has quit IRC | 10:36 | |
| *** avishayb has joined #openstack-infra | 10:37 | |
| *** guohliu has quit IRC | 10:40 | |
| *** markmcclain has joined #openstack-infra | 10:43 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Adding six dependency https://review.openstack.org/59235 | 10:44 |
| *** sarob has quit IRC | 10:51 | |
| *** rongze has joined #openstack-infra | 10:55 | |
| *** jishaomei has quit IRC | 10:56 | |
| *** adalbas has joined #openstack-infra | 10:57 | |
| *** che-arne has joined #openstack-infra | 10:58 | |
| *** salv-orlando has joined #openstack-infra | 10:59 | |
| *** jorisroovers has joined #openstack-infra | 11:00 | |
| *** yaguang has quit IRC | 11:02 | |
| *** rfolco has joined #openstack-infra | 11:07 | |
| *** pcm_ has joined #openstack-infra | 11:07 | |
| *** liang has quit IRC | 11:08 | |
| *** pcm_ has quit IRC | 11:10 | |
| *** pcm_ has joined #openstack-infra | 11:10 | |
| *** sdague_ has joined #openstack-infra | 11:11 | |
| *** jorisroovers has quit IRC | 11:11 | |
| *** wenlock has quit IRC | 11:13 | |
| *** DinaBelova has quit IRC | 11:14 | |
| *** sarob has joined #openstack-infra | 11:18 | |
| *** matsuhashi has quit IRC | 11:22 | |
| *** ArxCruz has joined #openstack-infra | 11:24 | |
| sdague | jog0: did you create an lp or somewhere for tracking er bugs / work? | 11:24 |
| *** sarob has quit IRC | 11:27 | |
| *** ljjjustin has quit IRC | 11:29 | |
| *** HenryG has joined #openstack-infra | 11:35 | |
| *** nati_ueno has joined #openstack-infra | 11:42 | |
| *** michchap_ has joined #openstack-infra | 11:42 | |
| *** michchap has quit IRC | 11:44 | |
| *** nprivalova has joined #openstack-infra | 11:49 | |
| *** nprivalova is now known as nadya_ | 11:50 | |
| *** jorisroovers has joined #openstack-infra | 11:53 | |
| *** aardvark has joined #openstack-infra | 11:53 | |
| *** yassine has quit IRC | 11:55 | |
| *** dtroyer_zz has joined #openstack-infra | 11:56 | |
| *** zigo_ has joined #openstack-infra | 11:56 | |
| *** nprivalova has joined #openstack-infra | 11:56 | |
| nadya_ | hi guys! I faced with a problem with nova+memcache. Tokens are marked as unauthorized in memcache. Does anyone know about this problem? | 11:57 |
| sdague | nadya_: that's really a #openstack question | 11:57 |
| *** cyeoh has quit IRC | 11:57 | |
| *** dtroyer has quit IRC | 11:57 | |
| *** zigo has quit IRC | 11:57 | |
| *** WarrenUsui has quit IRC | 11:57 | |
| *** cyeoh has joined #openstack-infra | 11:57 | |
| *** nadya_ has quit IRC | 11:57 | |
| *** sdake_ has quit IRC | 11:57 | |
| *** sdake_ has joined #openstack-infra | 11:57 | |
| *** markmc has quit IRC | 11:57 | |
| *** sdake_ has quit IRC | 11:58 | |
| *** sdake_ has joined #openstack-infra | 11:58 | |
| *** markmc has joined #openstack-infra | 11:58 | |
| *** dstanek_afk has joined #openstack-infra | 12:00 | |
| *** dstanek_afk has quit IRC | 12:04 | |
| *** markwash has quit IRC | 12:04 | |
| *** nprivalova is now known as nadya_ | 12:07 | |
| *** ruhe has joined #openstack-infra | 12:08 | |
| *** nati_ueno has quit IRC | 12:08 | |
| *** hashar has quit IRC | 12:16 | |
| *** sarob has joined #openstack-infra | 12:18 | |
| *** markwash has joined #openstack-infra | 12:18 | |
| *** pcm_ has quit IRC | 12:22 | |
| *** nosnos has quit IRC | 12:22 | |
| *** sarob has quit IRC | 12:23 | |
| *** rongze has quit IRC | 12:23 | |
| *** hashar has joined #openstack-infra | 12:25 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 12:26 | |
| *** heyongli has joined #openstack-infra | 12:32 | |
| *** smarcet has joined #openstack-infra | 12:36 | |
| *** saschpe_ has quit IRC | 12:41 | |
| *** saschpe has joined #openstack-infra | 12:43 | |
| *** fifieldt has quit IRC | 12:47 | |
| *** che-arne has quit IRC | 12:52 | |
| *** matsuhashi has joined #openstack-infra | 12:57 | |
| *** CaptTofu has joined #openstack-infra | 12:57 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Add gate-solum-devstack job https://review.openstack.org/57098 | 13:00 |
| *** pcm_ has joined #openstack-infra | 13:02 | |
| portante | good day, folks: is there somebody that could help track down why this swift container server did not start during this commit's last check job? https://review.openstack.org/#/c/53270/ | 13:02 |
| portante | what log file has the screen command where the container server is started? | 13:02 |
| portante | thanks | 13:03 |
| *** pcm_ has quit IRC | 13:03 | |
| *** pcm_ has joined #openstack-infra | 13:03 | |
| *** matsuhashi has quit IRC | 13:06 | |
| *** matsuhashi has joined #openstack-infra | 13:07 | |
| *** dkranz has quit IRC | 13:07 | |
| *** sandywalsh has joined #openstack-infra | 13:09 | |
| *** heyongli has quit IRC | 13:09 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: E128 really isn't sane. https://review.openstack.org/58902 | 13:10 |
| *** matsuhashi has quit IRC | 13:11 | |
| *** matsuhashi has joined #openstack-infra | 13:13 | |
| sdague | fungi: once you wake up I'm going to come after you with some reviews to look at, because in order to make a couple of switch overs I've got 3 config reviews that need to land with 2 er changes interleaved | 13:14 |
| *** matsuhashi has quit IRC | 13:17 | |
| *** che-arne has joined #openstack-infra | 13:17 | |
| *** pcm_ has quit IRC | 13:18 | |
| *** sarob has joined #openstack-infra | 13:18 | |
| *** rongze has joined #openstack-infra | 13:25 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:26 | |
| *** pcm_ has joined #openstack-infra | 13:28 | |
| *** pcm_ has joined #openstack-infra | 13:29 | |
| *** nati_ueno has joined #openstack-infra | 13:32 | |
| *** wenlock has joined #openstack-infra | 13:34 | |
| *** xchu has joined #openstack-infra | 13:36 | |
| *** pcm_ has quit IRC | 13:42 | |
| *** mancdaz has joined #openstack-infra | 13:44 | |
| *** ciypro is now known as chandankumar | 13:44 | |
| *** thomasem has joined #openstack-infra | 13:45 | |
| *** CaptTofu has quit IRC | 13:46 | |
| *** CaptTofu has joined #openstack-infra | 13:47 | |
| *** ruhe is now known as ruhe_ | 13:47 | |
| *** sarob has quit IRC | 13:50 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/config: Add oslo.messaging to release status tracking https://review.openstack.org/59374 | 13:51 |
| *** ruhe_ is now known as ruhe | 13:54 | |
| *** dkranz has joined #openstack-infra | 13:54 | |
| *** yamahata_ has joined #openstack-infra | 13:55 | |
| *** larsbutler has quit IRC | 13:56 | |
| *** nati_ueno has quit IRC | 13:57 | |
| *** alexpilotti has joined #openstack-infra | 13:57 | |
| *** nadya_ has quit IRC | 14:00 | |
| *** bpokorny has joined #openstack-infra | 14:01 | |
| *** jcoufal has quit IRC | 14:01 | |
| *** avishayb has quit IRC | 14:03 | |
| *** CaptTofu has quit IRC | 14:03 | |
| *** CaptTofu has joined #openstack-infra | 14:04 | |
| *** chandankumar is now known as ciypro | 14:05 | |
| *** julim has joined #openstack-infra | 14:06 | |
| *** marun has quit IRC | 14:08 | |
| *** marun has joined #openstack-infra | 14:09 | |
| *** nprivalova has joined #openstack-infra | 14:09 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add gate-climate-devstack https://review.openstack.org/58702 | 14:10 |
| *** dolphm has joined #openstack-infra | 14:11 | |
| *** mkulkin has left #openstack-infra | 14:12 | |
| *** mriedem has joined #openstack-infra | 14:12 | |
| *** pcm_ has joined #openstack-infra | 14:15 | |
| *** nati_ueno has joined #openstack-infra | 14:16 | |
| *** pcm_ has quit IRC | 14:17 | |
| *** pcm_ has joined #openstack-infra | 14:18 | |
| *** ruhe is now known as ruhe_ | 14:18 | |
| *** reaper has quit IRC | 14:18 | |
| *** sarob has joined #openstack-infra | 14:18 | |
| *** ruhe_ is now known as ruhe | 14:20 | |
| *** mfer has joined #openstack-infra | 14:20 | |
| *** mfer has quit IRC | 14:20 | |
| *** mfer has joined #openstack-infra | 14:21 | |
| *** dprince has joined #openstack-infra | 14:21 | |
| *** sarob has quit IRC | 14:23 | |
| *** yaguang has joined #openstack-infra | 14:24 | |
| *** jergerber has joined #openstack-infra | 14:29 | |
| *** weshay has joined #openstack-infra | 14:31 | |
| *** marun has quit IRC | 14:32 | |
| *** marun has joined #openstack-infra | 14:32 | |
| sdague | hey jog0's awake again :) now I can hound him a little on reviews | 14:33 |
| * jog0 ducks | 14:33 | |
| sdague | basically I just want to ensure I didn't screw up anything too bad so I can land through this set of swizzles and refactoring quickly (especially as I want to not land new recheck bugs until we get over to the directory) | 14:35 |
| sdague | and so when infra folks are awake, I can coordinate lands with them | 14:36 |
| *** avishayb has joined #openstack-infra | 14:37 | |
| jog0 | sdague: yeah, normally I don't like being harassed for reviews too much, but this is a good exception to that rule | 14:37 |
| fungi | sdague: hollow threats | 14:37 |
| sdague | fungi: :) | 14:37 |
| *** dcramer_ has joined #openstack-infra | 14:37 | |
| fungi | looks like jishaomei and lianglin have disappeared, but i'm going to guess that the jenkins master which has gate-tempest-python27 configured on it either isn't running the jenkins-gearman plugin or can't communicate via gearman protocol with the gearman server zuul is using | 14:40 |
| portante | clarkb, jog0: can I get some help debugging why swift container server did not start on a given job https://review.openstack.org/#/c/53270/, and how we can fix devstack to fail faster when that happens? | 14:41 |
| *** alexpilotti has quit IRC | 14:42 | |
| fungi | Adri2000: reed may be able to point you in the right direction, but the activity site isn't developed by the infra team (some company named bitergia) | 14:42 |
| fungi | katyafervent: what project in gerrit are you trying to create a branch in? it's probably just an acl permission you need added on it | 14:43 |
| fungi | portante: interesting... i see a log for it under http://logs.openstack.org/70/53270/11/check/check-grenade-devstack-vm/36429b6/logs/old/ but not http://logs.openstack.org/70/53270/11/check/check-grenade-devstack-vm/36429b6/logs/new/ (i wonder if it's missing from the devstack/grenade service list in new?) | 14:48 |
| portante | fungi: yes, the devstack.log file? | 14:49 |
| portante | If that is the one you are talking about, I have noticed that for a while | 14:49 |
| fungi | portante: well, in the old environment there's a screen-s-container.txt.gz but not in new | 14:50 |
| portante | yes | 14:50 |
| *** jcoufal has joined #openstack-infra | 14:50 | |
| sdague | mordred: well with no other known avenue to get support on the membership service, I'm going to tweet about it | 14:51 |
| sdague | ... a lot | 14:51 |
| mordred | sdague: please do | 14:51 |
| fungi | s-container is definitely in the services list in both old and new | 14:51 |
| fungi | sdague: tweet in all-caps | 14:51 |
| sdague | https://twitter.com/sdague/status/407522190792216576 | 14:51 |
| mordred | sdague: if you would like to tweet about the fact that infra runs a collaborative and scalable development infrastructure | 14:51 |
| fungi | also, the wiki for the www site says contact todd@openstack.org for support | 14:51 |
| portante | fungi: where is the script output for the new servies? | 14:51 |
| *** marun has quit IRC | 14:51 | |
| mordred | sdague: and that it doesn't make any sense for the website and the memberdata systems to be there | 14:52 |
| mordred | to not be there | 14:52 |
| sdague | agreed | 14:52 |
| fungi | portante: still looking. i was just comparing the localrc between them first to be sure | 14:52 |
| portante | fungi: k thanks | 14:52 |
| sdague | mordred: is there some sort of TC level statement we could make to help fight that fight? | 14:53 |
| sdague | because my foundation email the last time went no where | 14:53 |
| fungi | sdague: mordred: isn't there an assertion statement in the governance repo being voted on about that? | 14:53 |
| mordred | fungi: there is, but that has not been submitted to the list for discussion yet | 14:54 |
| mordred | we're still going back and forth on language :) | 14:54 |
| fungi | ahh, i was only kinda lurking the last tc meeting | 14:54 |
| *** jpeeler has joined #openstack-infra | 14:54 | |
| *** jpeeler has joined #openstack-infra | 14:54 | |
| mordred | sdague: https://review.openstack.org/#/c/58072 | 14:55 |
| sdague | mordred: awesome | 14:56 |
| *** nati_ueno has quit IRC | 14:56 | |
| mordred | sdague: ok - updated to address typo feedback | 14:56 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/devstack-gate: Up the default concurrency on tempest runs https://review.openstack.org/58605 | 14:57 |
| *** dhellmann is now known as dhellmann_ | 14:58 | |
| *** loq_mac has joined #openstack-infra | 14:58 | |
| sdague | so, https://review.openstack.org/#/c/59293/1 is a change that we need in infra before we can split up the queries.yaml into a directory of files | 14:58 |
| sdague | if anyone has a moment | 14:59 |
| *** loq_mac has quit IRC | 15:01 | |
| *** dstanek_afk has joined #openstack-infra | 15:03 | |
| jog0 | sdague: btw the issues dkranz is hitting in the gating on errors is scary | 15:07 |
| sdague | jog0: agreed, that's why we want to enforce it | 15:07 |
| jog0 | sdague: I think my takeaway from it is: ceilometer needs to up there QA game | 15:08 |
| jog0 | and they may need some friendly pushing to get there | 15:09 |
| sdague | yeh, ceilometer is kind of a mess | 15:10 |
| mordred | yeah - I was just thinking that if he's having too hard a time making the whitelist, that we should just start enforcing what we have right now | 15:10 |
| sdague | mordred: yeh, that's the plan | 15:10 |
| mordred | awesome | 15:10 |
| *** alcabrera has joined #openstack-infra | 15:10 | |
| mordred | I saw something about adding a * to ceilometer because they were adding them too quickly | 15:11 |
| sdague | dkranz: is going to push enforcing today | 15:11 |
| sdague | so, honestly, ceilometer is probably in a worse state than neutron | 15:11 |
| sdague | however, neutron I think it more important to get up to snuff | 15:11 |
| sdague | so neutron in icehouse, ceilometer in j | 15:11 |
| *** pcm_ has left #openstack-infra | 15:12 | |
| sdague | or consider de-integrating ceilometer and putting it back to an incubation state | 15:12 |
| portante | sdague: is this is the "fail gate jobs if ERROR messages encountered?" | 15:12 |
| mordred | sdague: above done, btw | 15:12 |
| sdague | portante: yes | 15:12 |
| portante | k thanks | 15:12 |
| sdague | we're using a white list, so if there are valid reasons for the errors, we can have them | 15:13 |
| sdague | the problem is ceilometer is adding new errors so fast you can't whitelist them | 15:13 |
| mordred | sdague: are there really valid reasons for errors? | 15:13 |
| sdague | mordred: I'm not sure yet | 15:13 |
| mordred | sdague: k. fair enough | 15:13 |
| sdague | however, I'm not going to say for 100% that there aren't | 15:14 |
| portante | mordred: if you are testing how error conditions are handled, perhaps? | 15:14 |
| mordred | my hypothesis is that a traceback in a log means that a bug happened | 15:14 |
| mordred | portante: don't think so - errors are different than exceptions | 15:14 |
| sdague | and the whitelist model at least lets us have that discussion | 15:14 |
| *** alexpilotti has joined #openstack-infra | 15:14 | |
| openstackgerrit | A change was merged to openstack-infra/config: also link in /etc/elastic-recheck/queries https://review.openstack.org/59293 | 15:14 |
| mordred | portante: an _exception_ caused by something runnins api commands means there is a bug in the code | 15:14 |
| * jd__ hangs his head in shame | 15:14 | |
| mordred | and error message printed to the logs mean that an error occured | 15:14 |
| *** rwsu has joined #openstack-infra | 15:14 | |
| mordred | but - I believe sdague madkes a great point, and it's entirely possible I'm wrong on that | 15:16 |
| portante | mordred: not convinced yet, as this seems to be about definition of ERROR and how it is used currently across projects | 15:16 |
| dkranz | mordred, sdague : So my issue is that fail on error is ready to go but I don';t think dev folk are really aware of it even with all the messages sent about it. | 15:16 |
| jd__ | dkranz: could you add a few links to logs with the errors in the bug report you reported btw? | 15:16 |
| dkranz | I would like to send a joint qa/infra announcement that this is being turned on | 15:16 |
| dkranz | jd__: I can. I saw your comments in the recent ticket. | 15:16 |
| fungi | portante: so... on the grenade failure for 53270, i can see in the syslog for the change you asked about that at 09:08:21 "container-server UNCAUGHT EXCEPTION..." | 15:17 |
| jd__ | dkranz: thanks; FWIW I've added this subject to our next meeting agenda | 15:17 |
| sdague | portante: right, it will help shape that conversation as well | 15:17 |
| dkranz | jd__: There is a possibility these errors came from a patch that was not yet merged | 15:17 |
| sdague | dkranz: so, honestly, just turn it on | 15:17 |
| mordred | agree | 15:17 |
| sdague | people have had plenty of warning | 15:17 |
| jd__ | dkranz: ok, that's what I think, that would come from the tempest tests we are pushing and that are not merged yet | 15:17 |
| mordred | they'll learn when they add new exceptions into the logs | 15:17 |
| *** dolphm has quit IRC | 15:17 | |
| dkranz | So be it. | 15:17 |
| portante | mordred, sdague, dkranz, at the least wouldn't we want to fail jobs with exceptions (python tracebacks emitted) | 15:18 |
| *** xchu has quit IRC | 15:18 | |
| mordred | portante: I'm not sure what you mean? | 15:18 |
| portante | fungi: looking | 15:18 |
| *** sarob has joined #openstack-infra | 15:18 | |
| mordred | portante: (we're talking about being able to trap that exceptions emitted in logs should trigger a test failure - but to do that, it means that in normal successful operation, we need to have no exceptions in the log) | 15:19 |
| portante | agreed | 15:19 |
| fungi | portante: unfortunately the 4k limit we put on syslog messages truncated it | 15:19 |
| mordred | fungi: doh | 15:19 |
| dkranz | sdague: https://review.openstack.org/#/c/58848/ | 15:20 |
| portante | yes, was that with old or new code? | 15:20 |
| *** dkliban has joined #openstack-infra | 15:20 | |
| mordred | dkranz: "except neutron" | 15:20 |
| fungi | portante: i think that's the old code. i see a couple minutes later in the syslog where container-server reports shutting down, but then i never see it mention starting again | 15:20 |
| portante | I looks like the old code, which in grizzly, and possibly havana, did not handle the signal sent to kill the servers | 15:20 |
| mordred | dkranz: sigh | 15:20 |
| fungi | portante: okay, so not relevant then. i'll keep staring | 15:21 |
| dkranz | mordred: Do you have a better suggestion for now? | 15:21 |
| mordred | dkranz: nope. just sigh that it's needed - not sigh on you | 15:21 |
| portante | fungi: yes, but where is the command that starts the container seraver with the new code? | 15:21 |
| dkranz | mordred: I agree. This is clearly in the better late than never category | 15:22 |
| mordred | yah | 15:22 |
| portante | fungi: in the "old" directory there is a devstack.log.gz, do you know why there isn't one of those in the "new" directory? | 15:23 |
| *** reaper has joined #openstack-infra | 15:23 | |
| *** ryanpetrello has joined #openstack-infra | 15:26 | |
| fungi | portante: i could speculate, but the qa people probably have a lot more insight into grenade's inner workings there | 15:27 |
| fungi | portante: in the console.html i do see... | 15:27 |
| fungi | 2013-12-02 09:09:08.525 | + screen -S stack -p s-container -X stuff 'cd /opt/stack/new/swift && /opt/stack/new/swift/bin/swift-container-server /etc/swift/container-server/1.conf -v || echo "s-container failed to start" | tee "/opt/stack/new/status/stack/s-container.failure"^M' | 15:27 |
| portante | yes, and I would like to konw if the s-container.failure exists? | 15:29 |
| portante | . | 15:29 |
| fungi | so it looks like it at least thinks it started, since i don't see the failure message echoed | 15:29 |
| fungi | but it doesn't seem to have created a screen-s-container.log for new | 15:30 |
| fungi | portante: i think maybe the grenade shell log is the equivalent of the devstack log for new | 15:31 |
| *** CaptTofu has quit IRC | 15:32 | |
| fungi | sdague: is that ^ an accurate statement? | 15:32 |
| *** dolphm has joined #openstack-infra | 15:33 | |
| ryanpetrello | so Monty merged the "gate pecan against Ceilometer and others" patch, but it looks like the ceilometer tests are failing trying to connect to mongo: http://logs.openstack.org/43/58643/1/check/gate-pecan-tox-ceilometer-stable/68bc334/console.html.gz | 15:33 |
| ryanpetrello | anybody have ideas about this? figured I'd ask rather than just running a recheck | 15:33 |
| *** jcoufal has quit IRC | 15:33 | |
| *** loq_mac has joined #openstack-infra | 15:34 | |
| mordred | ryanpetrello: I blame ... uhm ... | 15:35 |
| mordred | ryanpetrello: (I do not) | 15:35 |
| fungi | (EMONGO) | 15:35 |
| mordred | ryanpetrello: I do agree with you that it looks like it's failing connecting to mongo | 15:35 |
| jd__ | let's blame mordred since he merged the patch, no? | 15:35 |
| * jd__ runs | 15:35 | |
| ryanpetrello | haha | 15:35 |
| *** loquacities has joined #openstack-infra | 15:36 | |
| *** CaptTofu has joined #openstack-infra | 15:36 | |
| mordred | do we have mongo runing on unit test slaves now? | 15:36 |
| ryanpetrello | yea, I was wondering if I was missing something that was causing this to run in a place where it shouldn't | 15:36 |
| *** adalbas has quit IRC | 15:36 | |
| jd__ | mordred: no but Ceilometer's run-tests.sh launches it temporarily | 15:36 |
| mordred | oh - I see that. neat | 15:36 |
| jd__ | while we're talking about that, is there a PostgreSQL and MySQL instances running on the slaves that can be used? | 15:37 |
| mordred | jd__, ryanpetrello: it looks lke it started up to me | 15:37 |
| mordred | jd__: yes there is | 15:37 |
| *** dhellmann_ is now known as dhellmann | 15:37 | |
| mordred | jd__: nova uses it in their unit tests | 15:37 |
| jd__ | mordred: I want to do that in Ceilometer too | 15:38 |
| mordred | jd__: I want you to do that in ceilometer! :) | 15:38 |
| *** loq_mac has quit IRC | 15:38 | |
| jd__ | mordred: is there a documentation on the URL we shall use or should I grep nova? | 15:38 |
| *** datsun180b has joined #openstack-infra | 15:39 | |
| *** datsun180b is now known as datsun_F40PH | 15:40 | |
| mordred | jd__: user openstack_citest passwd openstack_citest schema openstack_citest :) | 15:40 |
| jd__ | noted, thanks! | 15:40 |
| mordred | jd__: but you may want to look at nova/tests/db/test_migrations.py | 15:40 |
| mordred | jd__: it's got the testing to see if mysql/postgres are there and whatnot | 15:40 |
| *** wenlock_ has joined #openstack-infra | 15:40 | |
| jd__ | nice | 15:41 |
| *** sparkycollier has joined #openstack-infra | 15:42 | |
| fungi | ahh the ol' alma mater... my brother was supposed to give his undergrad research presentation this morning, but they had to cancel the entire symposium because a raccoon crawled into the power substation and shorted it out, so the entire campus is running on emergency generators | 15:42 |
| fungi | hillbilly-u | 15:42 |
| *** dims has joined #openstack-infra | 15:43 | |
| ryanpetrello | fungi: only counts as hillbilly-u if they ate the raccoon afterwards | 15:43 |
| anteaya | ha ha ha | 15:44 |
| *** rcleere has joined #openstack-infra | 15:44 | |
| ryanpetrello | so re: the mongo ceilometer issue | 15:44 |
| ryanpetrello | any ideas? is anybody against me doing a recheck? | 15:45 |
| *** markwash has quit IRC | 15:46 | |
| jog0 | ryanpetrello: if its a bug that isn't in your code, file a bug first and then ok to recheck bug x | 15:47 |
| *** pcrews has joined #openstack-infra | 15:48 | |
| ryanpetrello | same procedure if there's a known bug involved that's already been fixed? | 15:48 |
| *** adalbas has joined #openstack-infra | 15:48 | |
| ryanpetrello | and how's it work if the bug is in *another* launchpad project? | 15:48 |
| *** senk has joined #openstack-infra | 15:48 | |
| *** dolphm has quit IRC | 15:50 | |
| jog0 | ryanpetrello: it jut works if its another launchhpad project | 15:50 |
| jog0 | ryanpetrello: the bug has been fixed? | 15:50 |
| ryanpetrello | in the case of one failure, yes | 15:51 |
| *** sarob has quit IRC | 15:51 | |
| *** jroovers has joined #openstack-infra | 15:52 | |
| *** sparkycollier has quit IRC | 15:53 | |
| *** dolphm has joined #openstack-infra | 15:54 | |
| *** jorisroovers has quit IRC | 15:55 | |
| *** rnirmal has joined #openstack-infra | 15:55 | |
| *** loquacities has quit IRC | 15:55 | |
| *** jorisroovers has joined #openstack-infra | 15:55 | |
| *** ^d has joined #openstack-infra | 15:56 | |
| *** jroovers has quit IRC | 15:57 | |
| *** briancline has quit IRC | 15:58 | |
| *** SergeyLukjanov has quit IRC | 15:58 | |
| fungi | portante: sdague: i'm at a loss for why no /opt/stack/new/screen-logs/screen-s-container.log | 15:58 |
| fungi | was archived in http://logs.openstack.org/70/53270/11/check/check-grenade-devstack-vm/36429b6/console.html | 15:58 |
| *** dkranz has quit IRC | 15:58 | |
| sdague | fungi: did it get that far? | 15:58 |
| fungi | sdague: yep, should have been created. it got as far as stuffing the command to start the container service into the screen session | 15:59 |
| fungi | by then there should have been a screen log for that session | 15:59 |
| fungi | right? | 15:59 |
| *** mgagne has joined #openstack-infra | 16:00 | |
| *** mgagne has quit IRC | 16:00 | |
| *** mgagne has joined #openstack-infra | 16:00 | |
| fungi | there is later indication during testing that the container service was not actually running, and no mention in syslog that it was started at all after grenade upgraded it | 16:00 |
| *** SergeyLukjanov has joined #openstack-infra | 16:01 | |
| sdague | so, we did experience straight up screen races before | 16:01 |
| sdague | that's the only thing I can think of with this | 16:02 |
| fungi | how did we work around that? i know at one point there was a suggestion that we simply disable running under screen in ci, but concerns were expressed that would deviate too far from what a typical dev runs | 16:02 |
| *** DinaBelova has joined #openstack-infra | 16:03 | |
| sdague | http://logs.openstack.org/70/53270/11/check/check-grenade-devstack-vm/36429b6/console.html#_2013-12-02_09_09_08_525 | 16:03 |
| sdague | fungi: honestly, I don't remember atm | 16:03 |
| *** jorisroovers has quit IRC | 16:03 | |
| *** DinaBelova has quit IRC | 16:03 | |
| fungi | yup, that's the line i was looking at | 16:03 |
| *** luisg has joined #openstack-infra | 16:04 | |
| *** jcooley_ has joined #openstack-infra | 16:04 | |
| jeblair | good morning | 16:04 |
| fungi | morning, jeblair | 16:04 |
| *** SergeyLukjanov has quit IRC | 16:05 | |
| *** mancdaz has quit IRC | 16:06 | |
| jog0 | sdague: quick opinion question on https://review.openstack.org/#/c/59090 | 16:06 |
| jog0 | then Iwill review your patches | 16:06 |
| sdague | shoot | 16:06 |
| jog0 | https://review.openstack.org/#/c/59090/7/doc/source/index.rst | 16:06 |
| jog0 | that rule wasn't what was initially discussed on the ML therad | 16:07 |
| jog0 | thread* | 16:07 |
| jog0 | and although I like that rule and am ++ on it, do we need to get ML feedback on that? | 16:07 |
| *** mgagne has quit IRC | 16:08 | |
| zaro | mordred, jeblair : w00t! WIP vote merged! | 16:08 |
| jog0 | as that goes a little further then the no license header needed in an file with nothing else in it | 16:08 |
| sdague | so do yuo see it as materially different than the thread? | 16:08 |
| *** mgagne has joined #openstack-infra | 16:08 | |
| *** mgagne has quit IRC | 16:08 | |
| *** mgagne has joined #openstack-infra | 16:08 | |
| jog0 | "empty files shouldn't | 16:08 |
| jog0 | > have license headers, because their content of emptiness isn't | 16:08 |
| jog0 | > copyrightable" | 16:08 |
| jog0 | I see that as different | 16:08 |
| sdague | ok, fair | 16:08 |
| jog0 | although the only case I Found in nova was nova/network/security_group/__init__.py | 16:09 |
| mgagne | hashar: would you mind rebasing this one? https://review.openstack.org/#/c/56397/ | 16:09 |
| sdague | do you know of any actual hits in our source trees that would fail one versio the other? | 16:09 |
| jog0 | which is pretty trivial | 16:09 |
| *** pete5 has joined #openstack-infra | 16:09 | |
| jog0 | let me try another project | 16:09 |
| fungi | sdague: portante: i guess if you see it again, recheck bug 1183449 ? | 16:09 |
| uvirtbot | Launchpad bug 1183449 in grenade "Screen is causing non-deterministic behaviour" [High,New] https://launchpad.net/bugs/1183449 | 16:09 |
| openstackgerrit | Julien Danjou proposed a change to openstack-dev/hacking: Update pep8 and flake8 requirements https://review.openstack.org/59152 | 16:09 |
| jog0 | jd__: thanks btw ^^ | 16:10 |
| jd__ | jog0: np :) | 16:10 |
| jog0 | so we are discussing your other patch, which I like the idea of but ... see scrollback | 16:11 |
| ttx | fungi, mordred: would be nice to have https://review.openstack.org/#/c/58616/ accepted before EOD tomorrow, since we'll need to cut some savanna mp branches then | 16:12 |
| *** ywu has joined #openstack-infra | 16:13 | |
| *** ciypro has quit IRC | 16:13 | |
| hashar | mgagne: yeah still have to resurrect the cloverphp publisher change | 16:13 |
| ttx | fungi: thx! | 16:13 |
| *** dkranz has joined #openstack-infra | 16:13 | |
| hashar | mgagne: aka https://review.openstack.org/#/c/52940/3 | 16:13 |
| mgagne | hashar: mind rebasing on master instead? | 16:13 |
| *** nprivalova has quit IRC | 16:14 | |
| mgagne | hashar: your change will be able to get merged without cloverphp which is unrelated | 16:14 |
| hashar | true | 16:14 |
| hashar | :D | 16:14 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: raise JenkinsJobsException instead of Exception https://review.openstack.org/56397 | 16:16 |
| hashar | mgagne: done | 16:16 |
| hashar | no more Exception :] | 16:16 |
| mgagne | :D | 16:16 |
| *** pabelanger_ has joined #openstack-infra | 16:17 | |
| hashar | the other patch, I will resurect it whenever I have time | 16:17 |
| *** sarob has joined #openstack-infra | 16:18 | |
| *** pabelanger_ has quit IRC | 16:18 | |
| *** pabelanger_ has joined #openstack-infra | 16:18 | |
| *** pabelanger has quit IRC | 16:18 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add missed milestone-proposed rule https://review.openstack.org/58616 | 16:19 |
| *** pabelanger has joined #openstack-infra | 16:19 | |
| *** pabelanger has quit IRC | 16:19 | |
| *** pabelanger_ is now known as pabelanger | 16:19 | |
| *** MIDENN_ has joined #openstack-infra | 16:20 | |
| *** pabelanger_ has joined #openstack-infra | 16:20 | |
| portante | fungi thanks, will do | 16:21 |
| *** pabelanger has quit IRC | 16:21 | |
| *** pabelanger has joined #openstack-infra | 16:22 | |
| *** mrodden has joined #openstack-infra | 16:22 | |
| *** MIDENN_ has quit IRC | 16:22 | |
| EmilienM | could someone from core have a review on https://review.openstack.org/#/c/59247/ ? it's for adding puppet-ironic into stackforge | 16:23 |
| *** mdenny has quit IRC | 16:23 | |
| fungi | zaro: congrats on the gerrit wip patch! | 16:25 |
| *** loq_mac has joined #openstack-infra | 16:25 | |
| ttx | mordred: around? | 16:26 |
| zaro | fungi: thx. almost gave up on it. | 16:27 |
| *** mdenny has joined #openstack-infra | 16:27 | |
| *** loquacities has joined #openstack-infra | 16:28 | |
| *** pblaho has quit IRC | 16:28 | |
| *** pblaho has joined #openstack-infra | 16:28 | |
| *** loq_mac has quit IRC | 16:29 | |
| anteaya | zaro: well done on getting your patch into upstream gerrit | 16:29 |
| anteaya | yay you | 16:29 |
| anteaya | that alone might be worthy of a talk :D | 16:30 |
| zaro | maybe just on the politics of upstream gerrit. :) | 16:30 |
| *** markmc has quit IRC | 16:30 | |
| pleia2 | good morning | 16:31 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add a zuul client https://review.openstack.org/58841 | 16:31 |
| anteaya | zaro: yeah, I think you would have good attendance | 16:32 |
| anteaya | morning pleia2 | 16:32 |
| anteaya | read the scrollback on last weeks meeing | 16:32 |
| anteaya | sorry I wasn't here for the puppetboard item | 16:32 |
| anteaya | you covered it well | 16:33 |
| *** adalbas has quit IRC | 16:33 | |
| anteaya | I'm probably going to be absent from this week's meeting too :( | 16:33 |
| *** jcooley_ has quit IRC | 16:33 | |
| *** yaguang has quit IRC | 16:33 | |
| *** sarob has quit IRC | 16:33 | |
| *** dstanek_afk has quit IRC | 16:36 | |
| *** hashar has quit IRC | 16:37 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Add barbican.json project file https://review.openstack.org/59476 | 16:37 |
| jog0 | sdague: I still don't see the resolved_at keyword logic | 16:37 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add barbican.json project file https://review.openstack.org/59476 | 16:37 |
| jog0 | in https://review.openstack.org/#/c/59252 | 16:37 |
| sdague | jog0: it isn't there yet | 16:38 |
| jog0 | sdague: I thought this was ready to review | 16:38 |
| sdague | that was going to come in after the switch over | 16:38 |
| jog0 | sdague: why? | 16:38 |
| sdague | because sorting out the logic of what we want wasn't yet decided | 16:38 |
| sdague | I can go the other path and remove references to it | 16:39 |
| *** senk has quit IRC | 16:39 | |
| sdague | I guess we crossed wires about what the state of this patch would be | 16:39 |
| jog0 | sdague: ahh, yeah I would prefer you mention that the lgoic isn't fully sortreed out and is coming soon | 16:39 |
| *** mancdaz has joined #openstack-infra | 16:39 | |
| jog0 | maybe just an update to the readme and commig msg | 16:39 |
| sdague | ok, can do | 16:40 |
| jog0 | cool | 16:40 |
| *** danger_fo_away is now known as dangers | 16:40 | |
| *** avishayb has quit IRC | 16:42 | |
| *** mancdaz has quit IRC | 16:42 | |
| *** mancdaz has joined #openstack-infra | 16:43 | |
| pleia2 | so much email | 16:43 |
| openstackgerrit | Justin Shepherd proposed a change to openstack-infra/config: Adding chef-style-rubocop job https://review.openstack.org/59478 | 16:43 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: Fix E122,E126,E128 items in codebase https://review.openstack.org/59272 | 16:44 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: move queries.yaml into a queries subdir https://review.openstack.org/59252 | 16:44 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: method extraction for readability https://review.openstack.org/59263 | 16:44 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: time to grow up and fit in 80 columns https://review.openstack.org/59262 | 16:44 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: wrapping the README.rst file to 80 cols https://review.openstack.org/59261 | 16:44 |
| sdague | jog0: updated | 16:44 |
| jog0 | sdague: now i have to go and +2 all your paches again ... sigh | 16:45 |
| sdague | jog0: well, you're the the one that wanted the change :) | 16:45 |
| sdague | I thought the +2 would come back | 16:45 |
| jog0 | sdague: some will I think | 16:45 |
| sdague | as they are rebases, except for the bottom one | 16:45 |
| pleia2 | all hail the rebase detection script | 16:46 |
| sdague | so let me know if you like the new 59252 | 16:46 |
| sdague | as fungi and mordred merged the thing in config that needed to land before it | 16:46 |
| *** adalbas has joined #openstack-infra | 16:47 | |
| *** loquacities has quit IRC | 16:47 | |
| jog0 | so for https://review.openstack.org/#/c/59252 I assume you have actaully ran and tested your changes | 16:47 |
| jog0 | as our tset coverage in gate for this is pretty so-so | 16:47 |
| jog0 | especially for such a big change | 16:47 |
| jog0 | sdague: ^ | 16:47 |
| sdague | the check success scripts work and do the right thing | 16:47 |
| sdague | so the loader path is tested | 16:48 |
| *** loq_mac has joined #openstack-infra | 16:48 | |
| jog0 | that covers the changes here so cool | 16:49 |
| sdague | yep | 16:49 |
| *** DinaBelova has joined #openstack-infra | 16:49 | |
| *** senk has joined #openstack-infra | 16:51 | |
| ttx | mordred: nm | 16:51 |
| *** briancline has joined #openstack-infra | 16:52 | |
| *** gyee has joined #openstack-infra | 16:52 | |
| jog0 | sdague: dinner time so wil lbe AFK for a bit, all patches but the web one are reviewed | 16:53 |
| sdague | cool | 16:53 |
| jog0 | that one on a high level looks good to me, but haven't dug in yet. also jim should review that one too | 16:53 |
| sdague | yep, agreed | 16:53 |
| sdague | and that's orthoginal enough | 16:54 |
| jog0 | yup | 16:54 |
| jog0 | have fun dealing with the new and imroved no stacktrace gate | 16:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix pep8 errors in modules dir https://review.openstack.org/47644 | 16:55 |
| sdague | ok, mtreinish did a +A on the dir change one | 16:55 |
| *** mrodden has quit IRC | 16:55 | |
| *** rongze has quit IRC | 16:56 | |
| *** pabelanger has quit IRC | 16:56 | |
| sdague | which means we might actually need a manual restart of that bot... IIRC it doesn't do code changes CD | 16:56 |
| sdague | then https://review.openstack.org/#/c/59301/1 is next up | 16:56 |
| sdague | to move the graphs over to it | 16:57 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: move queries.yaml into a queries subdir https://review.openstack.org/59252 | 16:57 |
| mordred | sdague: ok- e-r waterfall of style patches above shoudl all fall in | 16:57 |
| sdague | mordred: coolio | 16:57 |
| mordred | sdague: I also did not review anything complex or that did anything substantial | 16:58 |
| sdague | yep | 16:58 |
| sdague | the patch series was called cleanup for a reason :) | 16:58 |
| openstackgerrit | Rossella Sblendido proposed a change to openstack-infra/devstack-gate: Test neutron full stack https://review.openstack.org/59484 | 16:58 |
| *** dkranz has quit IRC | 16:59 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: wrapping the README.rst file to 80 cols https://review.openstack.org/59261 | 16:59 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: time to grow up and fit in 80 columns https://review.openstack.org/59262 | 16:59 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: method extraction for readability https://review.openstack.org/59263 | 16:59 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Fix E122,E126,E128 items in codebase https://review.openstack.org/59272 | 16:59 |
| sdague | nice | 16:59 |
| sdague | ok, time to heat up some lunch | 17:00 |
| jeblair | mordred: let me know if you want to talk about the -2 i just left on https://review.openstack.org/#/c/48064/ | 17:01 |
| jeblair | mordred: i'm happy to help if you would like it | 17:01 |
| *** boris-42 has quit IRC | 17:01 | |
| *** dstanek_afk has joined #openstack-infra | 17:02 | |
| mordred | jeblair: morning! I believe that I have addressed your concerns, but it will take landing the patches that are listed as dependencies first before we can recheck that to verify that it is correct | 17:02 |
| fungi | ttx: added sergey as the initial savanna-milestone member (he doesn't seem to be around at the moment) | 17:03 |
| mordred | jeblair: I was intending on just pinging you here to talk about the 3-patch sequence anyway :) | 17:03 |
| *** mrodden has joined #openstack-infra | 17:03 | |
| jeblair | mordred: ok, i'll go look at the others (regardless, i don't think we should merge yours yet since it is not yet self-gating) | 17:04 |
| mordred | jeblair: I agree - I believe that once the other two land, this patch will be self-gating | 17:04 |
| mikal | So... Who understands tempest more than me? Or perhaps just testr? | 17:05 |
| jeblair | mordred: you might try replying to major structural comments in the future. :) | 17:05 |
| mikal | Does testr promise to only run tests from one test class on a single worker? | 17:05 |
| mikal | Or could tests from a single class be of different workers at the same time? | 17:05 |
| mordred | jeblair: bah. that would be way too communicative :) | 17:06 |
| mordred | mikal: could be on different workers | 17:06 |
| mikal | Hmmm | 17:06 |
| mikal | I'm looking at a tempest failure... | 17:06 |
| mikal | Its a delete instance that failed | 17:06 |
| mordred | mikal: testr makes no promises about adjacency or scheduling | 17:06 |
| mikal | Because the instance was deleted | 17:06 |
| mikal | So it feels a bit like a race in testr to me | 17:06 |
| mordred | mikal: nice. sounds like test interaction issues for sure | 17:06 |
| mordred | mikal: race in the tests | 17:07 |
| *** dstanek_afk has quit IRC | 17:07 | |
| mordred | not in testr | 17:07 |
| mikal | Well, becuse of tyhe way testr runs the tests | 17:07 |
| mordred | the tests are assuming a sequential ordering, most likely, and are not cleaning up / setting up properly | 17:07 |
| mordred | a test assuming and not creating a particular state is a bug in the test | 17:07 |
| jeblair | mordred: in https://review.openstack.org/#/c/59196/3 you say "Additionally, we need to stop updating | 17:07 |
| jeblair | requirements.txt itself. | 17:07 |
| jeblair | " | 17:07 |
| jeblair | mordred: i don't understand that and the patch doesn't seem to do anything about that. | 17:07 |
| mordred | jeblair: crappit - I believe that is an aborted leftover sentence. lemme fix | 17:08 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: new maven plugin jobs to build and deploy maven plugins to maven nexus https://review.openstack.org/46099 | 17:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack/requirements: Align projects list with mirror-needing projects https://review.openstack.org/59196 | 17:11 |
| mordred | jeblair: there ya go - the english in that commit message probably makes more sense | 17:11 |
| sdague | mikal: in testr we are running parallel only at the class level | 17:11 |
| sdague | mikal: sorry, in tempest's use of testr | 17:11 |
| mikal | sdague: oh, interesting | 17:11 |
| mikal | sdague: so... I have a test that's failing because the instance is in an unexpected state | 17:12 |
| mikal | It looks to me like its trying to delete and instance which is already deleted | 17:12 |
| mikal | Which gives me the sads | 17:12 |
| fungi | mordred: 59196 says "openstack/requirements itself also does not use the mirror, so remove | 17:12 |
| fungi | it from the list." | 17:12 |
| sdague | mikal: you have the test fail? | 17:12 |
| fungi | which i think is not entirely accurate? | 17:12 |
| mikal | sdague: http://logs.openstack.org/99/55499/11/gate/gate-tempest-devstack-vm-full/5bffadf/console.html | 17:12 |
| mordred | fungi: it is accurate. we explicitly do not want changes to openstack/requirements to hit the mirror | 17:12 |
| *** dkranz has joined #openstack-infra | 17:13 | |
| mordred | because otherwise we can't test changes to it | 17:13 |
| fungi | mordred: if you look at the mirror selection logic in devstack-gate, we skip mirror selection for the requirements integration job when the project is openstack/requirements | 17:13 |
| mordred | that's right | 17:13 |
| fungi | mordred: but other devstack-gate jobs which happen to run for changes to openstack/requirements are not skipped | 17:13 |
| jeblair | mordred: ok, whole thing makes sense; leaving -2 until everything else merges; you may need to ping me to lift that. | 17:13 |
| mordred | fungi: that's right | 17:13 |
| fungi | er, rather, mirror overriding is not skipped | 17:14 |
| mordred | fungi: I'm not sure how that's different from what the above says? | 17:14 |
| mikal | sdague: http://paste.openstack.org/show/54301/ | 17:14 |
| fungi | i interpret your patch as "changes to openstack/requirements" don't use the openstack pypi mirror when being tested in devstack-gate | 17:15 |
| mordred | fungi: right. we can't ever use the mirror for changes to openstack/requirements | 17:15 |
| mordred | because they wouldn't work | 17:15 |
| sdague | mikal: so that test is sending a hard reboot | 17:15 |
| mordred | if we think we're using it now, we're wrong | 17:15 |
| fungi | except they do, so long as the job is not the requirements integration test | 17:15 |
| mordred | no. it's not possible | 17:15 |
| sdague | and expects the server to come back around to ACTIVE | 17:15 |
| mordred | a change to requirements can introduce a thing whic is not in the mirror | 17:15 |
| *** dstanek_afk has joined #openstack-infra | 17:15 | |
| mordred | that means it cannot, by definition, be constrained by the mirror | 17:15 |
| mordred | no matter what the job is | 17:16 |
| mordred | or? | 17:16 |
| sdague | it looks like the libvirt driver is racing, lost the server in some part of the transition, then left the server in the HARD_REBOOT state because it was confused | 17:16 |
| fungi | mordred: no, you're right | 17:16 |
| mordred | fungi: however, this conversation has just pointed out to me another potential failure | 17:16 |
| mordred | since requirements is in the integrated gate, changes behind it to non-requirements repos are going to want to be able to test in the context of that requirements change | 17:17 |
| fungi | mordred: i misread. we skip mirror selection when running any devstack-agte job for openstack/requirements and *also* when running the requirements-integration jobs for any project | 17:17 |
| mordred | fungi: ok. cool. I was worried that I fundamentally misunderstood something | 17:17 |
| mordred | fungi: there is a weird case I just listed above, but since it hasn't bitten us yet, perhaps it's not as devastating as one might think :) | 17:17 |
| fungi | yeah, i agree that's a hole | 17:18 |
| fungi | and i'm not immediately seeing an easy solution to it | 17:18 |
| mordred | nope. | 17:19 |
| sdague | that being said, we could create a new server for the reboot test, as the issue might be that something else there left it in a silly state | 17:19 |
| jeblair | mordred: in what context would they fail to test what was expected? | 17:19 |
| mordred | fungi: not unless we create temporary mirrors for requirements changes and let subsequent gate jobs figure out how to use that | 17:19 |
| *** dstanek_afk has quit IRC | 17:19 | |
| mordred | jeblair: if a requirements change introduces a requirement that cannot be satisfied by the mirror | 17:20 |
| jeblair | mordred: (i understand the situation you are describing -- i'm wondering _exactly what thing_ they want to test won't be there) | 17:20 |
| jeblair | mordred: what thing will want to test that? | 17:20 |
| jeblair | mordred: what change will be behind it that will require that? | 17:20 |
| mordred | it's not that | 17:20 |
| mordred | jeblair: then a subsequent change to nova should, by zuul combination rules, pull in that requirements change and then run update.py in devstack | 17:20 |
| mordred | which means that it should just fail | 17:20 |
| *** dstanek_afk has joined #openstack-infra | 17:21 | |
| jeblair | mordred: except that that change won't even pass check tests until _after_ the requirement change lands | 17:21 |
| fungi | mordred: so going back to the devstack-gate mirror selection logic for 59196, once the other two changes merge we should be able to simplify that conditional to just if [ "$DEVSTACK_GATE_REQS_INTEGRATION" -eq "0" ] | 17:21 |
| mordred | no - not a change to nova that wants to use that | 17:21 |
| mordred | jeblair: a change that's approved after the requirements change is approved | 17:21 |
| mordred | will pick up the requirements chagne in the devstack update.py invocation | 17:21 |
| jeblair | mordred: then it will have been approved with its check tests failing. | 17:21 |
| mordred | no | 17:21 |
| *** SergeyLukjanov has joined #openstack-infra | 17:21 | |
| mordred | it's not a nova change that contains that requirements change | 17:21 |
| sdague | mikal: actually, the test triggers a server rebuild if it's not active | 17:22 |
| mordred | its' a nova change that does not contain that requirements change - but is approved after it in sequence in the gate | 17:22 |
| sdague | so that logic should already be there | 17:22 |
| jeblair | mordred: oh, but the requiremnts change should not affect it -- the requirements change itself tests that | 17:22 |
| mordred | which means that the state of the requirements repo at the point the nova change is tested will be the state of the pending requirements change | 17:22 |
| mordred | and since devstack syncs the requirements files of all of the projects with the checked out requirements repo | 17:22 |
| jeblair | mordred: oh, i think i see what you're saying | 17:23 |
| *** senk has quit IRC | 17:23 | |
| mordred | jeblair: it's a REALLY REALLY odd sequencing issue | 17:23 |
| jeblair | mordred: it will force an unsatisfiable dependency on the unsuspecting nova change | 17:23 |
| mordred | yup | 17:23 |
| jeblair | mordred: i'm not sure it's that odd | 17:23 |
| jeblair | mordred: it sounds quite likely. | 17:23 |
| mordred | well, I mean, in practice, it's probably not likely to occur with great frequency | 17:23 |
| mordred | hrm. no - you're right, it shold happen more frequently - why are we not seeing the failure? | 17:24 |
| jeblair | mordred: won't it mean that any requirements change will completely break the gate until the mirror is updated? | 17:24 |
| mikal | Huh | 17:24 |
| *** thedodd has joined #openstack-infra | 17:24 | |
| mikal | This error isn't an error in the driver | 17:24 |
| mordred | only ones that introduce completely new boundaries that were previously not resolvable | 17:24 |
| mikal | Its a silly log level to use | 17:24 |
| mordred | new requirements arent' a problem because nothing will be using them yet | 17:25 |
| *** dstanek_afk has quit IRC | 17:25 | |
| *** pblaho has quit IRC | 17:25 | |
| mordred | new upper bounds probably usually work because they'll be constraining within the set of thigns we had already | 17:25 |
| jeblair | mordred: ah, ok. that may be the answer to the question then. | 17:25 |
| mordred | but if we had an upper bound, and we introduce a new change which raises and and also raises the lower bound past the previous upper bound | 17:25 |
| mordred | problem is - landing THAT change is going to be almost impossible :) | 17:26 |
| jeblair | mordred: we probably don't ever merge requirements changes that are not resolvable with the previous set -- that would after all, break cd. | 17:26 |
| mordred | yah | 17:26 |
| mordred | so - neat. actually,let's call it a feature | 17:26 |
| mordred | jeblair: how's that for a mental exercise to start your monday? | 17:26 |
| jeblair | mordred: ++ | 17:26 |
| fungi | though the iso8601>=0.1.8 change verges very close to that scenario | 17:26 |
| jeblair | mordred: well, back to clarkb's d-g refactor now. :) | 17:27 |
| mordred | fungi: yes. I believ ethe iso8601 landing is a bug | 17:27 |
| mordred | fungi: that should never have occured | 17:27 |
| *** moted has joined #openstack-infra | 17:27 | |
| *** dolphm has quit IRC | 17:27 | |
| mordred | and that we did not prevent it from landing indicates a hole in our testing | 17:27 |
| mordred | which hopefully we'll fix soon | 17:27 |
| *** mfer has quit IRC | 17:27 | |
| *** rongze has joined #openstack-infra | 17:28 | |
| sdague | mikal: well it actually is an error somewhere | 17:28 |
| sdague | because after a server reboot, the server isn't coming back | 17:28 |
| sdague | that's actually why tempest is failing you | 17:28 |
| sdague | if you hard reboot a server it should return to an active state | 17:28 |
| *** reed has joined #openstack-infra | 17:28 | |
| sdague | after some time, and we wait 196 seconds for it to do that | 17:29 |
| *** afazekas has quit IRC | 17:29 | |
| *** dstanek_afk has joined #openstack-infra | 17:29 | |
| reed | good day everybody | 17:30 |
| fungi | howdy reed | 17:30 |
| reed | how was the turkey? | 17:31 |
| *** loq_mac has quit IRC | 17:31 | |
| reed | mine was good, not wild this year... for the first time I could try the legs :) | 17:31 |
| *** senk has joined #openstack-infra | 17:32 | |
| *** dizquierdo has quit IRC | 17:32 | |
| *** mfer has joined #openstack-infra | 17:33 | |
| reed | speaking of serious business, how can we get to an agreement on https://review.openstack.org/#/c/53644/ ? | 17:34 |
| *** fbo is now known as fbo_away | 17:34 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-dev/hacking: Add a check for file with only comments https://review.openstack.org/59090 | 17:35 |
| *** hogepodge has joined #openstack-infra | 17:35 | |
| jeblair | ArxCruz: hi, i want to make sure i understand https://review.openstack.org/#/c/54386/ | 17:36 |
| ArxCruz | jeblair: yes? | 17:36 |
| jeblair | ArxCruz: you say kernel-headers is installed by "devstack_gate.sh"; do you mean by nodepool? | 17:36 |
| *** bpokorny1 has joined #openstack-infra | 17:37 | |
| ArxCruz | jeblair: yes | 17:37 |
| ArxCruz | Elizabeth test in her infra for both precise and fedora | 17:37 |
| jeblair | ArxCruz: so in ./modules/openstack_project/files/nodepool/scripts/prepare_devstack.sh | 17:37 |
| *** jpich has quit IRC | 17:37 | |
| *** bpokorny2 has joined #openstack-infra | 17:37 | |
| ArxCruz | jeblair: yup | 17:38 |
| ArxCruz | sudo yum -y install python-devel make automake gcc gcc-c++ kernel-devel redhat-lsb-core | 17:38 |
| ArxCruz | this line install the kernel-headers | 17:38 |
| *** bpokorny has quit IRC | 17:38 | |
| jeblair | ArxCruz, pleia2: i think what we want to do in the long run is make sure that devstack itself handles that... | 17:39 |
| pleia2 | ++ | 17:39 |
| jeblair | ArxCruz, pleia2: do you know if devstack itself handles any of those? | 17:39 |
| ArxCruz | jeblair: I can check, if don't I can submit a patch :) | 17:39 |
| pleia2 | haven't tested that yet, but I can spin something up this afternoon | 17:40 |
| pleia2 | or ArxCruz :) | 17:40 |
| ArxCruz | :) | 17:40 |
| jeblair | ArxCruz, pleia2: cool, so i think this should be what we aim for: | 17:40 |
| jeblair | ArxCruz, pleia2: 1) devstack itself installs as many openstack dependencies as it can | 17:40 |
| jeblair | ArxCruz, pleia2: 2) anything that devstack-gate needs in order for it to run that isn't installed in 3) should be in devstack-gate | 17:41 |
| jeblair | ArxCruz, pleia2: 3) anything that the nodepool setup scripts need to run should be installed by the nodepool setup scripts | 17:41 |
| *** bpokorny1 has quit IRC | 17:42 | |
| jeblair | ArxCruz, pleia2: so 2 and 3 should be as small as possible; ideally, nothing. :) | 17:42 |
| pleia2 | sounds good | 17:42 |
| jeblair | ArxCruz, pleia2: so i'll go ahead and +2 54386 then | 17:44 |
| *** adalbas has quit IRC | 17:44 | |
| ArxCruz | jeblair: beer++ for you :) | 17:44 |
| ArxCruz | :D | 17:44 |
| pleia2 | I have a DMV appointment to run off to, hopefully it won't take all day ;) | 17:44 |
| openstackgerrit | Justin Shepherd proposed a change to openstack-infra/config: Adding chef-style-rubocop job https://review.openstack.org/59478 | 17:49 |
| *** hogepodge has quit IRC | 17:50 | |
| ArxCruz | jeblair: whenever you have time, take a look at this one https://review.openstack.org/#/c/59473/ :) | 17:51 |
| *** UtahDave has joined #openstack-infra | 17:52 | |
| jeblair | mordred: -1 on https://review.openstack.org/#/c/59105/ (i note because it's blocking a critical bug) | 17:54 |
| mordred | jeblair: looking | 17:54 |
| mordred | jeblair: GAH. wow. how did I manage to mess up a change that small | 17:55 |
| jeblair | mordred: no "-Wall". :( | 17:55 |
| Alex_Gaynor | mordred: We've acquired a transitive dependency which doesn't have it's packages on pypi :( | 17:55 |
| mordred | Alex_Gaynor: dammit | 17:56 |
| fungi | oh, wow. and how did i miss that it lacked a format string argument to the replicate command? | 17:56 |
| mordred | Alex_Gaynor: which one? and from where? | 17:56 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/gerritlib: Add replicate command https://review.openstack.org/59105 | 17:56 |
| Alex_Gaynor | mordred: pytidylib, from sphinxcontrib-docbookrestapi- | 17:56 |
| mordred | Alex_Gaynor: I'm so sad | 17:56 |
| Alex_Gaynor | Me too :-( | 17:56 |
| *** adalbas has joined #openstack-infra | 17:58 | |
| openstackgerrit | A change was merged to openstack-infra/gerritlib: Add replicate command https://review.openstack.org/59105 | 17:58 |
| fungi | mordred: jeblair: we wanted to tag a release of gerritlib now ^ that's in, right? | 17:58 |
| mordred | fungi: I'd like to do that - but there were a couple of other changes in master since our last release that don't have anything to do with this command | 17:59 |
| *** alcabrera has quit IRC | 17:59 | |
| mordred | fungi, jeblair: none of them look scary though | 17:59 |
| *** alcabrera has joined #openstack-infra | 17:59 | |
| jeblair | mordred, fungi: agreed; 0.3.0 ? | 18:00 |
| *** nati_ueno has joined #openstack-infra | 18:00 | |
| mordred | jeblair: sounds great to me | 18:00 |
| jeblair | i will do this thing. | 18:00 |
| fungi | jeblair: yes, looks good | 18:00 |
| fungi | since it's not just bug fixes | 18:00 |
| *** wusui has joined #openstack-infra | 18:01 | |
| *** krotscheck has joined #openstack-infra | 18:01 | |
| fungi | if memory serves, we're consuming releases of gerritlib via pypi on review.o.o, so we can't land the related jeepyb changes until that happens | 18:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Added openstack-chef-repo to list https://review.openstack.org/58585 | 18:01 |
| jeblair | fungi, mordred: 0.3.0 pushed | 18:02 |
| *** aardvark has quit IRC | 18:02 | |
| mordred | jeblair: woot | 18:02 |
| *** markwash has joined #openstack-infra | 18:02 | |
| clarkb | morning | 18:02 |
| jeblair | clarkb: hi there | 18:02 |
| fungi | looks like pypi upload completed for it | 18:02 |
| fungi | yup, 0.3.0 is on pypi now | 18:03 |
| mordred | jeblair, fungi: please check me very carefully on the manage-projects changes - given that I messed up the gerritlib change :) | 18:03 |
| jeblair | mordred: have you tested them? | 18:03 |
| fungi | i tried to review them very carefully, but then again i messed up catching the error on that gerritlib change | 18:04 |
| mordred | jeblair: no. I'm not really sure how to test the things like the github failure (it's the logic about failure and resume I'm concerned about most) | 18:04 |
| mordred | jeblair: I'd love feedback or ideas on a good way to test that | 18:04 |
| jeblair | mordred: right, but have you tested that it still runs and does the right thing under normal circumstances? | 18:05 |
| reed | fungi, thanks for helping with the openid server patch... need to get that one moving | 18:05 |
| *** pabelanger has joined #openstack-infra | 18:05 | |
| *** pabelanger has quit IRC | 18:05 | |
| *** pabelanger has joined #openstack-infra | 18:05 | |
| fungi | reed: no problem. testing it now to see if it's still functionally broken | 18:06 |
| clarkb | so much scrollback | 18:06 |
| jeblair | mordred: i take it that's a no as well. :( | 18:07 |
| *** senk has quit IRC | 18:07 | |
| mordred | jeblair: no. I have not. let me go do that real quick (sorry, just got back to real internets last night - interactive remote things now possible) | 18:07 |
| *** branen has quit IRC | 18:09 | |
| mordred | anybody mind if I stop puppet on review-dev ? | 18:09 |
| *** senk has joined #openstack-infra | 18:10 | |
| fungi | mordred: not i | 18:10 |
| jeblair | mordred: go for it | 18:10 |
| *** mestery_ has joined #openstack-infra | 18:11 | |
| *** mestery has quit IRC | 18:11 | |
| *** senk has quit IRC | 18:13 | |
| clarkb | ArxCruz: did you see I had to roll back the change to add fedora hosts to nodepool? looked like permissions issues | 18:14 |
| ArxCruz | clarkb: no I didn't | 18:14 |
| ArxCruz | clarkb: what's happening ? | 18:15 |
| *** senk has joined #openstack-infra | 18:15 | |
| clarkb | ArxCruz: at some point we try to write to /opt (a fiel copy iirc) and that fails | 18:15 |
| ArxCruz | clarkb: but that's related to my patch? I don't remember touch anything related to /opt | 18:15 |
| mordred | jeblair: hrm. has manage-projects ever worked on review-dev ? | 18:16 |
| clarkb | ArxCruz: its related in that your patch wasn't sufficient to make it go on fedora | 18:16 |
| clarkb | ArxCruz: we need more bootstrapping somewhere | 18:16 |
| jeblair | mordred: i think you did your initial testing there | 18:16 |
| *** dhellmann is now known as dhellmann_ | 18:16 | |
| mordred | jeblair: heh. ok | 18:16 |
| clarkb | ArxCruz: I am digging up the stacktrace now | 18:16 |
| ArxCruz | clarkb: okay, if you need help, let me know | 18:17 |
| mordred | jeblair: I am now getting key errors :) | 18:17 |
| ArxCruz | clarkb: I'm leaving now, but i'll be back in a few hours, have english class :) | 18:17 |
| openstackgerrit | A change was merged to openstack-infra/config: Pass $mysql_password through, part two https://review.openstack.org/58618 | 18:17 |
| clarkb | ArxCruz: np have fun | 18:17 |
| *** openstackgerrit has quit IRC | 18:18 | |
| *** vipul is now known as vipul-away | 18:18 | |
| *** vipul-away is now known as vipul | 18:18 | |
| *** openstackgerrit has joined #openstack-infra | 18:18 | |
| clarkb | ArxCruz: sudo: sorry, you must have a tty to run sudo <- I think that may be the problem | 18:19 |
| fungi | clarkb: ArxCruz: yes, ssh -t usually | 18:19 |
| ArxCruz | clarkb: oh... I remember that in my Fedora image I change the sudoers to accept tty... | 18:19 |
| clarkb | fungi: I am guessing paramiko is given a tty without asking on ubuntu but not on fedora? | 18:19 |
| *** sarob has joined #openstack-infra | 18:19 | |
| fungi | quite possible | 18:19 |
| ArxCruz | clarkb: Defaults requiretty | 18:20 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added config options to not overwrite jobs desc https://review.openstack.org/52080 | 18:20 |
| *** wusui has quit IRC | 18:20 | |
| ArxCruz | clarkb: so, you want me to make the change and resubmit or a new patch? | 18:20 |
| *** wusui has joined #openstack-infra | 18:21 | |
| clarkb | ArxCruz: is that in the sudoers file? I don't think we want to edit the sudoers file | 18:21 |
| clarkb | ArxCruz: instead paramiko should request a tty | 18:21 |
| ArxCruz | ssh -t works ? | 18:21 |
| clarkb | ArxCruz: yeah the equivalent of ssh -t in nodepool with paramiko | 18:21 |
| ArxCruz | clarkb: can't remember, but it seems paramiko doesn't have this support | 18:22 |
| fungi | ArxCruz: http://www.lag.net/paramiko/docs/paramiko.SSHClient-class.html#invoke_shell | 18:22 |
| *** hogepodge has joined #openstack-infra | 18:22 | |
| ArxCruz | fungi: thanks I will check | 18:22 |
| ArxCruz | clarkb: what's the change # ? | 18:22 |
| *** mestery_ is now known as mestery | 18:23 | |
| ArxCruz | have to go now, ttyl | 18:23 |
| clarkb | ArxCruz: this will be a different change | 18:23 |
| clarkb | the reverted change was to openstack-infra/config | 18:24 |
| fungi | ArxCruz: though poking around more, you may just want the get_pty() method of teh session object | 18:24 |
| ArxCruz | clarkb: weird, it was just a yum install patch | 18:24 |
| jeblair | mordred: non-urgent: i left an idea on https://review.openstack.org/#/c/58770/ that i think you should evaluate | 18:24 |
| mordred | jeblair: yes. I agree with you - and I have thoughts in that area | 18:25 |
| *** ArxCruz has quit IRC | 18:25 | |
| dprince | pleia2: question for you on infra access to TripleO nodes. Do we currently have (or plan to have) a way for infra to publically shell into the slaves, etc? | 18:26 |
| dprince | pleia2: Is the only public access for infra managed through the overcloud API accessible for the nodepool? | 18:27 |
| mordred | jeblair: https://github.com/emontyoncrack/wookie18 | 18:28 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Do github last https://review.openstack.org/59107 | 18:28 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Catch errors centrally, and bail out more https://review.openstack.org/59106 | 18:28 |
| mordred | only found one tiny thing | 18:28 |
| jeblair | mordred: w00t | 18:28 |
| clarkb | is wookie18 testing the new manage-projects? | 18:29 |
| mordred | clarkb: yes | 18:29 |
| mordred | clarkb: github project created. gerrit project created. repo populated in gerrit. local mirror populated | 18:29 |
| mordred | acls applied | 18:29 |
| mordred | so - best I can tell, it's no worse than current manage-projects | 18:29 |
| clarkb | woot | 18:30 |
| mordred | it MAY be more resilient - but until github craps us, it's tough to say with 100% surity | 18:30 |
| clarkb | mordred: you could do something sinister like put an extry for github.com in /etc/hosts | 18:32 |
| clarkb | so that "github" fails | 18:32 |
| mordred | clarkb: speaking of - review-dev.openstack.org is in /etc/hosts on review-dev and the entry is wrong | 18:32 |
| clarkb | that is awesome | 18:32 |
| jeblair | mordred, clarkb, fungi: so once that is all merged in, should we also start requiring new projects have 2 patches: first create the repo, then jjb/zuul? | 18:33 |
| jeblair | (to avoid the git.o.o/zuul race) | 18:33 |
| clarkb | jeblair: so that I am double sure. That race is where jobs can be started by zuul prior to content being in git.o.o which is where jobs expect the repos to be | 18:34 |
| mordred | right | 18:35 |
| jeblair | clarkb: that's my understanding from your description. :) i think you said when that happens you have to rm-rf the git repo on zuul | 18:35 |
| mordred | ultimatley having a salt trigger that says "run puppet on git.o.o, then on review.o.o, then on zuul.o.o would be neat | 18:36 |
| clarkb | jeblair: no, that race is different | 18:36 |
| mordred | but we do not have that yet | 18:36 |
| clarkb | jeblair: zuul isn't cloning from git.o.o, it is cloning from review.o.o. The problem was when manage-projects created an empty repo on review.o.o without pushing any content to it | 18:36 |
| jeblair | clarkb: oooh. | 18:36 |
| jeblair | clarkb: that only happened because of github failures? | 18:37 |
| jeblair | clarkb: so, ideally, fixed by mordred's change as well? | 18:37 |
| clarkb | jeblair: and it seemed to be doing that because github would fail between those two steps causing manage-projects to move to the next project | 18:37 |
| clarkb | jeblair: yup | 18:37 |
| clarkb | I don't think we need split changes yet. Would be good to see if the situation is much better after mordred's change without extra hand holding | 18:37 |
| mordred | clarkb: ++ | 18:37 |
| jeblair | clarkb, mordred, fungi: ok, so based on that; we should be able to handle single-change creation patches, and we expect mordreds fixes to fix all known creation problems? | 18:38 |
| clarkb | I still need to review it, but yes :) | 18:38 |
| mordred | clarkb: enjoy the review | 18:39 |
| mordred | clarkb: I split it into three pieces, just for you! | 18:39 |
| sdague | so can someone restart the openstackrecheck bot? it's not clear to me that it took the code updates | 18:39 |
| fungi | jeblair: clarkb: mordred: that's my take on it, yes | 18:40 |
| fungi | sdague: sure, just a sec and i'll check it out before i restart it too | 18:40 |
| sdague | fungi: yeh, any log files would be appreciated | 18:40 |
| clarkb | mordred: thank you, I will look at it now since it seems to be the topic of discussion | 18:40 |
| *** sandywalsh has quit IRC | 18:40 | |
| clarkb | jeblair: fungi: mordred: should I approve things if they look good or do we want to do more testing first? | 18:40 |
| jeblair | clarkb: i think we can aprv jeepyb changes | 18:41 |
| fungi | clarkb: agreed, though i'm re-reviewing the one which just got a new patchset | 18:41 |
| *** rongze has quit IRC | 18:42 | |
| fungi | yup, all three lgtm | 18:43 |
| *** senk has quit IRC | 18:43 | |
| clarkb | ok, I will review quickly and hit +A if they look good (I expect them to) | 18:43 |
| *** hdd has joined #openstack-infra | 18:44 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/jenkins-job-builder: Enable ignore_cache flag on jenkins_jobs.ini https://review.openstack.org/59515 | 18:45 |
| *** osanchez has quit IRC | 18:47 | |
| *** osanchez has joined #openstack-infra | 18:49 | |
| jeblair | sdague: do we actually want to optionally set configdrive, or just always use metadata? https://review.openstack.org/#/c/59303/1 | 18:50 |
| *** dolphm has joined #openstack-infra | 18:50 | |
| sdague | jeblair: so... it feels like we should be able to test both | 18:53 |
| jeblair | mordred: one more thing on https://review.openstack.org/#/c/59107/ | 18:53 |
| sdague | fungi: so I figured out the issue | 18:53 |
| jeblair | sdague: ok, but are we actually going to test both in the gate? | 18:53 |
| mtreinish | sdague: well the way we've do it now is metadata | 18:54 |
| mtreinish | and then have specific config drive tests | 18:54 |
| fungi | sdague: oh? | 18:54 |
| mtreinish | that way we test both in the gate | 18:54 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: move to elastic-recheck web console in share https://review.openstack.org/59294 | 18:54 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: use queries directory instead of file for elastic-recheck https://review.openstack.org/59301 | 18:54 |
| sdague | mtreinish: actually, you changed the devstack default to config drive | 18:55 |
| sdague | so we are always doing config drive | 18:55 |
| sdague | we could just revert that I suppose | 18:55 |
| sdague | fungi: https://review.openstack.org/59301 | 18:55 |
| sdague | so I missed one place in config that the queries dir was specified | 18:55 |
| *** luisg has quit IRC | 18:56 | |
| mtreinish | sdague: I did? Ok, cool | 18:56 |
| fungi | sdague: yeah, i just started looking at it... seems there are two processes running, one started on the 15th and the other started when that last change merged | 18:56 |
| *** luisg has joined #openstack-infra | 18:56 | |
| sdague | fungi: interesting | 18:56 |
| clarkb | mordred: I agree with jeblair's -1. We should do that to avoid confusion | 18:56 |
| fungi | (or shortly thereafter... 17:08) | 18:56 |
| *** osanchez has quit IRC | 18:56 | |
| *** luisg has quit IRC | 18:56 | |
| *** luisg has joined #openstack-infra | 18:57 | |
| *** loq_mac has joined #openstack-infra | 18:57 | |
| clarkb | fungi: nice catch on the grenade defaults thing in layout.yaml | 18:57 |
| *** sandywalsh has joined #openstack-infra | 18:58 | |
| *** sarob has quit IRC | 18:58 | |
| sdague | mtreinish: oh, sorry it was dprince | 18:58 |
| clarkb | jeblair: fungi: If I fix that really quick should we take the plunge and push that change through? note it does make the job names much longer | 18:58 |
| *** sarob has joined #openstack-infra | 18:58 | |
| dkranz | clarkb: Is it possible to anything like a regexp match in a logstach query for a "message? | 18:58 |
| sdague | https://github.com/openstack-dev/devstack/commit/24f6efadbdef558655abc5f1052bb61ae87b55f0 | 18:59 |
| clarkb | dkranz: no regexes, but you can do globs and simpler things like that. http://lucene.apache.org/core/2_9_4/queryparsersyntax.html | 18:59 |
| jeblair | clarkb: if the system is busy (it is), that would be safer as a 2-part change (jjb, then zuul). | 18:59 |
| dkranz | clarkb: Thanks. I want to write something that will parse the whitelist in tempest and check if the log error is still ocurring. | 19:00 |
| clarkb | jeblair: good point, I will split the change in two | 19:00 |
| jeblair | clarkb: i think part 1 will need to keep the old jobs in order to pass the layout.yaml check :( | 19:00 |
| clarkb | jeblair: hmm I don't like that | 19:01 |
| jeblair | clarkb: alternative: stop puppet on zuul until the jobs exist. | 19:02 |
| *** sarob has quit IRC | 19:02 | |
| *** derekh has quit IRC | 19:03 | |
| clarkb | jeblair: that avoids the problem. But will make the process much cleaner | 19:03 |
| clarkb | jeblair: I can go with that | 19:03 |
| sdague | fungi: so regardless, it won't work until - https://review.openstack.org/#/c/59301/2 is merged | 19:04 |
| sdague | because it's config is sending it down the wrong path | 19:04 |
| clarkb | jeblair: should I keep it in one change and stop puppet on zuul, or would you prefer we do the git dance? | 19:04 |
| fungi | sdague: agreed, i was just reviewing that one | 19:04 |
| sdague | but it is weird that there are zombie processes | 19:04 |
| jeblair | clarkb: i'm okay with one change and you stopping puppet on zuul while it goes in. | 19:05 |
| fungi | sdague: shall i go ahead and kill the old one? | 19:06 |
| jeblair | it's a big change | 19:06 |
| clarkb | jeblair: ok thanks, that simplifies a bunch of stuff | 19:06 |
| sdague | fungi: yep | 19:06 |
| *** loq_mac has quit IRC | 19:06 | |
| *** jcooley_ has joined #openstack-infra | 19:06 | |
| sdague | it's just spitting out messages that it doesn't find errors in channel | 19:06 |
| *** loq_mac has joined #openstack-infra | 19:06 | |
| *** che-arne has quit IRC | 19:07 | |
| fungi | sdague: kill both pids in that case? | 19:07 |
| *** nati_ueno has quit IRC | 19:07 | |
| clarkb | wow git wants me to gc my config repo | 19:08 |
| *** loq_mac has quit IRC | 19:08 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Create two classes of d-g jobs. https://review.openstack.org/58370 | 19:08 |
| clarkb | fungi: jeblair mordred ^ addresses fungi's comment | 19:09 |
| *** loq_mac has joined #openstack-infra | 19:09 | |
| openstackgerrit | A change was merged to openstack-infra/config: use queries directory instead of file for elastic-recheck https://review.openstack.org/59301 | 19:09 |
| *** loq_mac has quit IRC | 19:09 | |
| sdague | fungi: yes | 19:10 |
| *** loq_mac has joined #openstack-infra | 19:10 | |
| sdague | kill them all | 19:10 |
| *** nati_ueno has joined #openstack-infra | 19:10 | |
| sdague | let puppet sort it out :) | 19:10 |
| sdague | the graphs should come back 15 minutes after the merge as well, which will be nice | 19:11 |
| *** rongze has joined #openstack-infra | 19:12 | |
| *** dstanek_afk has quit IRC | 19:12 | |
| clarkb | jeblair: we seem to have lost mordred, I will just address your comment on 59107 then we can push those changes through | 19:12 |
| *** dstanek_afk has joined #openstack-infra | 19:13 | |
| *** jcooley_ has quit IRC | 19:13 | |
| *** loq_mac has quit IRC | 19:13 | |
| *** jcooley_ has joined #openstack-infra | 19:13 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/jeepyb: Do github last https://review.openstack.org/59107 | 19:14 |
| *** jcooley_ has quit IRC | 19:14 | |
| clarkb | jeblair: fungi ^ | 19:14 |
| mordred | clarkb: hey - sorry - I was filling out my board candidate application so that people could properly nominate me | 19:14 |
| *** fungi has quit IRC | 19:15 | |
| mordred | clarkb: thanks! | 19:15 |
| *** hashar has joined #openstack-infra | 19:15 | |
| ekarlso | any news on this mordred ? https://bugs.launchpad.net/openstack-ci/+bug/1242569 | 19:18 |
| uvirtbot | Launchpad bug 1242569 in openstack-ci "manage-projects error on new project creation" [Critical,In progress] | 19:18 |
| ekarlso | or clarkb | 19:18 |
| mordred | ekarlso: we're working on it | 19:18 |
| jeblair | ekarlso: if you read the bug you'll note the link to a change that fixes it | 19:18 |
| mordred | clarkb: btw - did you see the note that the new HP cloud region is now GA - so we shoudl be able to start using it | 19:18 |
| clarkb | clarkb: I saw your note to me about it, OWA is being cranky so no email today yet | 19:19 |
| clarkb | wow | 19:19 |
| *** rongze has quit IRC | 19:20 | |
| *** fungi has joined #openstack-infra | 19:20 | |
| *** loq_mac has joined #openstack-infra | 19:20 | |
| fungi | ugh. he.net is suddenly experiencing major ipv6 packet loss | 19:21 |
| ekarlso | fyi, it's GA in both West and East :) | 19:21 |
| ekarlso | @ clarkb and mordred | 19:21 |
| *** dhellmann_ is now known as dhellmann | 19:22 | |
| clarkb | I have approved the jeepyb changes | 19:22 |
| mordred | neat. | 19:23 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Refactor manage-projects for readability https://review.openstack.org/59084 | 19:23 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Catch errors centrally, and bail out more https://review.openstack.org/59106 | 19:23 |
| clarkb | fungi: if you rereview https://review.openstack.org/#/c/58370/ I can stop puppet on zuul.o.o and babysit that process (though packet loss may make that hard) | 19:23 |
| ekarlso | can you push that change also ? ;) | 19:23 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Do github last https://review.openstack.org/59107 | 19:23 |
| fungi | clarkb: you got it | 19:24 |
| *** nati_ueno has quit IRC | 19:24 | |
| *** nati_ueno has joined #openstack-infra | 19:24 | |
| clarkb | stopping puppet on zuul now | 19:25 |
| fungi | painful, painful packet loss but almost done clicking | 19:26 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Setup devstack-gate tests for Savanna https://review.openstack.org/57317 | 19:27 |
| fungi | the soap/xml thing or whatever the gerrit does becomes very flaky under packet loss | 19:27 |
| *** annegentle has quit IRC | 19:28 | |
| fungi | clarkb: approved (finally) | 19:28 |
| clarkb | woot thank you | 19:28 |
| *** loq_mac has quit IRC | 19:28 | |
| *** jcoufal has joined #openstack-infra | 19:29 | |
| fungi | sdague: fwiw, the old e-r daemon process was ignoring sigterm (which is probably why it hung around), but woke up on sighup and went away | 19:30 |
| fungi | sdague: and the other process was gone already, replaced by one which started 10 minutes ago when your latest change got applied | 19:30 |
| *** NobodyCam is now known as NobodyCam_afk | 19:31 | |
| openstackgerrit | A change was merged to openstack-infra/config: Create two classes of d-g jobs. https://review.openstack.org/58370 | 19:33 |
| *** DinaBelova has quit IRC | 19:33 | |
| mtreinish | clarkb: have you seen this failure before?: http://logs.openstack.org/05/58605/2/check/check-tempest-devstack-vm-neutron/b8761ca/console.html | 19:33 |
| clarkb | mtreinish: yes, that seems to happen when zuul is under a lot of load | 19:34 |
| *** markmcclain has quit IRC | 19:35 | |
| clarkb | apache can't serve the file for some reason. There is a bug to have d-g retry like we retry in gerrit git prep | 19:35 |
| clarkb | I Think someone is owrking on it | 19:35 |
| sdague | fungi: cool | 19:36 |
| *** dstanek_afk has quit IRC | 19:36 | |
| clarkb | mtreinish: let me find it | 19:36 |
| mtreinish | clarkb: ok, cool thanks | 19:36 |
| ekarlso | https://bugs.launchpad.net/openstack-ci/+bug/1242569 could that be approve ? | 19:37 |
| uvirtbot | Launchpad bug 1242569 in openstack-ci "manage-projects error on new project creation" [Critical,Fix committed] | 19:37 |
| ekarlso | ehm | 19:37 |
| ekarlso | wrong one ! | 19:37 |
| *** dolphm has quit IRC | 19:37 | |
| ekarlso | https://review.openstack.org/#/c/56927/ | 19:37 |
| *** _david_ has joined #openstack-infra | 19:38 | |
| sdague | fungi: are there any other log messages on the box? | 19:38 |
| clarkb | mtreinish: https://bugs.launchpad.net/openstack-ci/+bug/1255353 is the bug. I will update it to note that zuul has the same problem | 19:38 |
| uvirtbot | Launchpad bug 1255353 in openstack-ci "during gate job 'error: Unable to get pack index https://review.openstack.org/p/openstack-infra/devstack-gate/objects/pack/pack-*.idx'" [High,Triaged] | 19:38 |
| sdague | it still looks like the bot isn't parsing things | 19:38 |
| mtreinish | clarkb: thanks, I'll use that for recheck then | 19:39 |
| clarkb | ekarlso: once we have had a chance to double check that all of the new jeepyb stuff is in place we can review the new project changes | 19:40 |
| clarkb | sdague: have a particular change you expected it to report on? I can use that to find relevant logs | 19:41 |
| ekarlso | nice clarkb :) | 19:42 |
| *** dolphm has joined #openstack-infra | 19:42 | |
| fungi | sdague: you mean as far as logging why it wasn't killed? i think it may have not had a write lock on the log by then... trying to figure out if maybe it was still writing to one of the already-rotated logs instead | 19:44 |
| sdague | clarkb: hmmm... well it's reported twice, but those might actually be unknown issues | 19:44 |
| sdague | clarkb: ok, let me let it sit for a bit and see | 19:47 |
| fungi | sdague: yeah, interestingly the old daemon seemed to have closed and reopened yesterday's log file and continued overwriting that today, but since it doesn't log its pid in the messages it's hard to confirm for certain that's what i'm looking at | 19:48 |
| *** rongze has joined #openstack-infra | 19:49 | |
| sdague | fungi: odd | 19:49 |
| sdague | so we only have the new bot running now, right? | 19:49 |
| fungi | sdague: yes | 19:49 |
| fungi | sdague: one process, which started at 19:19 | 19:50 |
| clarkb | mordred: /opt/jeepyb is up to date on review.o.o but pip freeze reports an old version. Did you ahve to do anyhting on review-dev to make it happy? | 19:50 |
| *** beagles is now known as beagles_brb | 19:50 | |
| clarkb | mordred: looks like there are a bunch of jeepyb's installed to /usr/local/lib/python2.7/dist-packages | 19:51 |
| fungi | clarkb: i've assumed those are from before the entrypoints rework in pbr | 19:52 |
| mordred | clarkb: me goes to look | 19:52 |
| clarkb | jenkins01 has the new jenkins jobs, waiting for jenkins02 now | 19:53 |
| *** rongze has quit IRC | 19:53 | |
| *** rwsu has quit IRC | 19:55 | |
| notmyname | where's the bot config file (to set what alerts in channel)? | 19:56 |
| *** rwsu has joined #openstack-infra | 19:56 | |
| *** yolanda has quit IRC | 19:58 | |
| clarkb | notmyname: openstack-infra/config/modules/gerritbot/gerritbot_channel_config.yaml | 19:59 |
| clarkb | jenkins01 and jenkins02 have the new d-g jobs, but not jenkins.o.o looking at that now | 20:00 |
| *** annegentle has joined #openstack-infra | 20:00 | |
| clarkb | running puppet by hand there to see what JJB is doing | 20:02 |
| *** nati_uen_ has joined #openstack-infra | 20:02 | |
| clarkb | puppet runs without execing jjb, going to run it by hand | 20:03 |
| notmyname | is the gerritbot healthy? it's configured to report on something that I'm not seeing reports for. | 20:03 |
| clarkb | notmyname: what is it not reporting on? the -2 events are known to not work | 20:03 |
| notmyname | hmm...or maybe it's just slow | 20:06 |
| *** nati_ueno has quit IRC | 20:06 | |
| *** dprince has quit IRC | 20:06 | |
| *** loq_mac has joined #openstack-infra | 20:06 | |
| *** MarkAtwood has joined #openstack-infra | 20:06 | |
| *** mfer has quit IRC | 20:07 | |
| *** loq_mac has quit IRC | 20:07 | |
| *** loq_mac has joined #openstack-infra | 20:07 | |
| *** ^d has quit IRC | 20:08 | |
| clarkb | fungi: jeblair: mordred: the way puppet copies files into /etc/jenkins_jobs/configs allows for old files to remain present | 20:09 |
| clarkb | and those stale files may break JJB | 20:09 |
| *** ^d has joined #openstack-infra | 20:09 | |
| *** sarob has joined #openstack-infra | 20:09 | |
| *** sarob has quit IRC | 20:09 | |
| *** sarob has joined #openstack-infra | 20:09 | |
| clarkb | I am going to remove the stale files on jenkins.o.o and rerun JJB | 20:09 |
| clarkb | then clean up jenkins01 and jenkins02 | 20:10 |
| *** mfer has joined #openstack-infra | 20:10 | |
| *** agordeev has joined #openstack-infra | 20:10 | |
| *** vkozhukalov has joined #openstack-infra | 20:10 | |
| vkozhukalov | here | 20:10 |
| *** notmyname has quit IRC | 20:10 | |
| devananda | mordred / clarkb / dansmith - is tripleo-ci still active? | 20:11 |
| fungi | clarkb: oh, should we be directly managing that config directory via puppet instead? | 20:11 |
| *** fbo_away is now known as fbo | 20:12 | |
| clarkb | fungi: we are, but I think the way we are enforcing the recursive config doesn't remove files that don't exist on the puppet master | 20:12 |
| fungi | got it | 20:12 |
| devananda | woops, wrong dan -- meant to ping dan prince. who's not here? | 20:13 |
| *** notmyname has joined #openstack-infra | 20:14 | |
| *** ruhe is now known as ruhe_ | 20:15 | |
| *** ruhe_ is now known as ruhe | 20:15 | |
| clarkb | devananda: I have no idea | 20:16 |
| devananda | clarkb: bah. you know everything :) | 20:16 |
| *** esker has joined #openstack-infra | 20:21 | |
| *** mrmartin has joined #openstack-infra | 20:23 | |
| *** ruhe_ has joined #openstack-infra | 20:24 | |
| clarkb | fungi: jeblair: mordred: All three jenkins masters should be happy now, I will propose a puppet change to fix this via puppet in a bit | 20:24 |
| clarkb | also going to spot check that the jobs went in properly | 20:24 |
| mordred | clarkb: awesome | 20:24 |
| *** alexpilotti has quit IRC | 20:25 | |
| clarkb | they look correct to me | 20:26 |
| clarkb | going to grab lunch quickly then continue through the job stuff | 20:26 |
| *** ruhe has quit IRC | 20:26 | |
| clarkb | (puppet is still not running on zuul.o.o_ | 20:26 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Trap failures due to old easy_install https://review.openstack.org/59532 | 20:27 |
| clarkb | mordred: are you working to fix that pbr bug on windows? | 20:27 |
| mordred | clarkb: no - I just got one reported in #openstack-dev and the fix was easy so I popped it in | 20:27 |
| clarkb | I think that fix should go into the next release of pbr. Also, we can in theory run tests on windows, can't believe I just said that | 20:28 |
| mordred | (like, less work than writing down the details in a bug) | 20:28 |
| mordred | I agree | 20:28 |
| mordred | I sort of think that running some tests on windows might not be a _terrible_ idea for things like pbr - but it also involves more things that I don't understand, so I keep deferring actually thinking about it | 20:28 |
| *** dcramer_ has quit IRC | 20:29 | |
| fungi | we can run tests on windows? not in our infra presumably | 20:29 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Mention errors in service logs for irc messages https://review.openstack.org/59534 | 20:29 |
| *** nati_uen_ has quit IRC | 20:30 | |
| *** nati_ueno has joined #openstack-infra | 20:30 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Trap failures due to old easy_install https://review.openstack.org/59532 | 20:32 |
| mordred | fungi: clouds have windows guests (me feels unclean saying that) | 20:33 |
| fungi | i'm sure they do... do we manage those with puppet? nodepool? what's the thought there? (not that i'm seriously entertaining the idea, but...) | 20:34 |
| fungi | and then there's the question of whether some member company is stepping up to provide free licenses for that operating system so we could legally use it | 20:35 |
| mgagne | fungi: somewhere in my TODO list is puppetizing a windows jenkins slave... | 20:36 |
| fungi | frightening | 20:36 |
| mgagne | fungi: I'm looking forward to it as much as you | 20:36 |
| fungi | i have no doubt | 20:36 |
| mgagne | fungi: the challenge is mainly with packages/dependencies | 20:36 |
| *** hogepodge has quit IRC | 20:37 | |
| *** beagles_brb is now known as beagles | 20:39 | |
| *** Ryan_Lane has joined #openstack-infra | 20:41 | |
| clarkb | fungi puppet and I dont think we worry about license fees but could be wrong | 20:43 |
| jeblair | mordred, fungi, clarkb: I'm going to self-approve https://review.openstack.org/#/c/58888/1 unless someone else reviews it in the next 5 mins or so. | 20:43 |
| jeblair | it's blocking work on a production bug in nodepool, and i'm pretty sure it's older than many changes that have been reviewed today. :( | 20:44 |
| *** denis_makogon_ has joined #openstack-infra | 20:44 | |
| *** Ryan_Lane has quit IRC | 20:45 | |
| *** denis_makogon_ is now known as denis_makogon | 20:46 | |
| mordred | jeblair: +A | 20:47 |
| *** branen has joined #openstack-infra | 20:47 | |
| jeblair | clarkb: yeah, we worry about license fees. windows is not a target development platform for openstack. i don't expect that we'd consider spinning up windows test nodes unless that changes. | 20:48 |
| jeblair | mordred: thx | 20:49 |
| *** rongze has joined #openstack-infra | 20:49 | |
| *** sarob has quit IRC | 20:53 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Remove remote-directory from publishing in api-jobs https://review.openstack.org/59538 | 20:53 |
| *** sarob has joined #openstack-infra | 20:53 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-dev/hacking: Add noqa support for H201 (bare except) https://review.openstack.org/57334 | 20:53 |
| *** emagana has joined #openstack-infra | 20:54 | |
| *** rongze has quit IRC | 20:54 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Improve image logging https://review.openstack.org/58888 | 20:56 |
| *** sarob has quit IRC | 20:57 | |
| *** marun has joined #openstack-infra | 20:58 | |
| *** markmcclain has joined #openstack-infra | 20:59 | |
| openstackgerrit | A change was merged to openstack/requirements: Align projects list with mirror-needing projects https://review.openstack.org/59196 | 21:00 |
| fungi | just over three hours after it was approved ^ (things are starting to pile up in the gate again) | 21:00 |
| fungi | actually 3.5 hours after approval | 21:01 |
| *** ^d is now known as ^demon|lunch | 21:01 | |
| *** alcabrera has quit IRC | 21:01 | |
| jeblair | bug 1242569 seems to be happening a lot | 21:03 |
| uvirtbot | Launchpad bug 1242569 in openstack-ci "manage-projects error on new project creation" [Critical,Fix committed] https://launchpad.net/bugs/1242569 | 21:03 |
| jeblair | ga | 21:03 |
| jeblair | bug 1253896 | 21:03 |
| uvirtbot | Launchpad bug 1253896 in neutron "test_minimum_basic_scenario fails with SSHException: Error reading SSH protocol banner" [High,In progress] https://launchpad.net/bugs/1253896 | 21:03 |
| *** herndon_ has joined #openstack-infra | 21:05 | |
| mordred | jeblair: oh good. I was worried that 1242569 was an issue | 21:05 |
| *** ^demon|lunch has quit IRC | 21:05 | |
| jeblair | mordred: ETOOMANYTABS | 21:05 |
| *** ^demon|lunch has joined #openstack-infra | 21:06 | |
| lifeless | devananda: or here? | 21:08 |
| lifeless | :P | 21:08 |
| devananda | lifeless: all teh roomz! | 21:08 |
| devananda | lifeless: here is fine, too | 21:08 |
| lifeless | devananda: actually, I need to power IRC off for a little | 21:08 |
| lifeless | devananda: I need to move my IB card from the old server to the new server and do basic physical setup. I'll be 20-30m, then back. | 21:08 |
| lifeless | ciao! | 21:08 |
| *** lifeless has quit IRC | 21:08 | |
| devananda | lifeless: question is vaguely around which parts of the testing story are reusable for ironic, what state those are in, and how to plug in other folks (eg some mirantis devs) who want to contribute to ironic CI | 21:09 |
| devananda | bah | 21:09 |
| devananda | he'll be back | 21:09 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Mention errors in service logs for irc messages https://review.openstack.org/59534 | 21:10 |
| *** loq_mac has quit IRC | 21:10 | |
| *** bpokorny has joined #openstack-infra | 21:11 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 21:13 | |
| *** loq_mac has joined #openstack-infra | 21:14 | |
| *** nicedice has joined #openstack-infra | 21:15 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add zuul rack-refs cron https://review.openstack.org/59543 | 21:16 |
| *** branen_ has joined #openstack-infra | 21:16 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add zuul pack-refs cron https://review.openstack.org/59543 | 21:16 |
| *** jpeeler1 has joined #openstack-infra | 21:18 | |
| *** jcooley_ has joined #openstack-infra | 21:19 | |
| *** branen has quit IRC | 21:20 | |
| *** _david_ has quit IRC | 21:20 | |
| *** bpokorny2 has quit IRC | 21:20 | |
| *** dkranz has quit IRC | 21:20 | |
| *** hdd has quit IRC | 21:20 | |
| *** jpeeler has quit IRC | 21:20 | |
| clarkb | jeblair: fungi: ready for me to start puppet on zuul.o.o? | 21:20 |
| fungi | clarkb: any time is fine by me | 21:21 |
| jeblair | clarkb: wfm | 21:21 |
| clarkb | starting | 21:21 |
| *** loquacities has joined #openstack-infra | 21:22 | |
| *** dcramer_ has joined #openstack-infra | 21:22 | |
| mordred | O M G my review queue is evil | 21:23 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Purge the JJB config dir. https://review.openstack.org/59550 | 21:23 |
| clarkb | jeblair: fungi: mordred: ^ that will fix the JJB config thing I ran into earlier | 21:24 |
| mordred | clarkb: we probably want to do that everywhere we do recurse => true | 21:24 |
| clarkb | mordred: probably. I can do a quick grep for recurse => true and see | 21:25 |
| *** MarkAtwood has quit IRC | 21:25 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add zuul pack-refs cron https://review.openstack.org/59543 | 21:25 |
| *** jcooley_ has quit IRC | 21:25 | |
| *** loq_mac has quit IRC | 21:25 | |
| *** jcooley_ has joined #openstack-infra | 21:26 | |
| *** sarob has joined #openstack-infra | 21:26 | |
| *** jcooley_ has quit IRC | 21:26 | |
| *** dcramer_ has quit IRC | 21:26 | |
| *** _david_ has joined #openstack-infra | 21:26 | |
| clarkb | mordred: a few of them look safe, others definitely aren't | 21:27 |
| *** hdd has joined #openstack-infra | 21:27 | |
| *** dkranz has joined #openstack-infra | 21:27 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation https://review.openstack.org/48803 | 21:27 |
| *** dolphm has quit IRC | 21:30 | |
| *** dcramer_ has joined #openstack-infra | 21:30 | |
| openstackgerrit | A change was merged to openstack-infra/config: Purge the JJB config dir. https://review.openstack.org/59550 | 21:30 |
| *** smarcet has left #openstack-infra | 21:32 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Use purge option where appropriate. https://review.openstack.org/59553 | 21:33 |
| clarkb | mordred: ^ those seem safe. Some of the others do not (they are used to apply permissions but not actually change files in many cases) | 21:33 |
| *** jpeeler1 is now known as jpeeler | 21:33 | |
| *** jpeeler has quit IRC | 21:33 | |
| *** jpeeler has joined #openstack-infra | 21:33 | |
| clarkb | zuul has the new config | 21:34 |
| mordred | clarkb: fantastic | 21:34 |
| *** che-arne has joined #openstack-infra | 21:36 | |
| *** dolphm has joined #openstack-infra | 21:38 | |
| *** Ryan_Lane has joined #openstack-infra | 21:41 | |
| clarkb | http://logs.openstack.org/33/59533/1/check/check-swift-devstack-vm-functional-default/5c62b7a/logs/devstack-gate-setup-host.txt.gz any idea why the pydistutils.cfg file cannot be created? that job failed and I am trying to figure out why | 21:41 |
| clarkb | http://logs.openstack.org/33/59533/1/check/check-swift-devstack-vm-functional-default/5c62b7a/logs/devstack-gate-setup-workspace-new.txt.gz is where the actual failure occured | 21:42 |
| *** pabelanger has quit IRC | 21:43 | |
| openstackgerrit | Anne Gentle proposed a change to openstack-infra/config: Changing the output directory that gets picked up for the API Quick Start https://review.openstack.org/59556 | 21:44 |
| clarkb | gah I think I see the problem | 21:45 |
| *** Ryan_Lane has quit IRC | 21:46 | |
| *** vkozhukalov has quit IRC | 21:47 | |
| portante | clarkb: have you seen this Syntax Error before fetching stuff from raw.github.com? http://logs.openstack.org/00/58500/4/check/check-swift-devstack-vm-functional/252a8d2/console.html | 21:47 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Set BRANCH_OVERRIDE in swift functional tests. https://review.openstack.org/59557 | 21:47 |
| *** weshay has quit IRC | 21:47 | |
| clarkb | jeblair: mordred: fungi: ^ can that get a quick review? I am going to make the change by hand on the jenkins masters really quick | 21:48 |
| *** vkozhukalov has joined #openstack-infra | 21:48 | |
| notmyname | clarkb: what does that do? | 21:48 |
| *** loquacities has quit IRC | 21:48 | |
| clarkb | notmyname: we did a massive overhaul of the jobs to make them handle releases more easily | 21:49 |
| jeblair | clarkb: +2 aprv at will | 21:49 |
| clarkb | notmyname: that allows us to override the branch for branch specific tests, but I forgot to put it in on the swift tests... | 21:49 |
| fungi | clarkb: safe to approve? | 21:49 |
| notmyname | clarkb: ok. so all non-master tests would fail? | 21:49 |
| clarkb | notmyname: currently all of them are failing | 21:49 |
| clarkb | fungi: ya I approved | 21:50 |
| fungi | clarkb: okay, well lgtm | 21:50 |
| *** loq_mac has joined #openstack-infra | 21:50 | |
| portante | clarkb: nevermind, https://bugs.launchpad.net/devstack/+bug/1257079 | 21:50 |
| uvirtbot | Launchpad bug 1257079 in devstack "Fails occured with the retieval of devstack/files/get-pip.py in gate" [Undecided,New] | 21:50 |
| *** sarob has quit IRC | 21:51 | |
| *** rongze has joined #openstack-infra | 21:51 | |
| *** adalbas has quit IRC | 21:51 | |
| *** sarob has joined #openstack-infra | 21:51 | |
| notmyname | clarkb: is that ^^ the bug to use for rechecks? | 21:51 |
| clarkb | notmyname: no, that is a different bug | 21:51 |
| clarkb | notmyname: let me fix this really quick to limit damage then I will write a bug | 21:52 |
| *** agordeev has quit IRC | 21:52 | |
| *** lifeless has joined #openstack-infra | 21:53 | |
| lifeless | devananda: ok, so hi | 21:53 |
| devananda | lifeless: question is vaguely around which parts of the testing story are reusable for ironic, what state those are in, and how to plug in other folks (eg some mirantis devs) who want to contribute to ironic CI | 21:53 |
| *** vkozhukalov has quit IRC | 21:54 | |
| lifeless | ok so | 21:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Set BRANCH_OVERRIDE in swift functional tests. https://review.openstack.org/59557 | 21:54 |
| lifeless | we have several different parallel efforts | 21:54 |
| lifeless | lets talk one at time, because though they have common bits, they are different enough that talking all at once is confusing | 21:55 |
| lifeless | there's the emulated baremetal stuff | 21:55 |
| lifeless | and there's actual baremetal stuff | 21:55 |
| *** sarob has quit IRC | 21:55 | |
| lifeless | emulated baremetal is coming along very well | 21:56 |
| *** dizquierdo has joined #openstack-infra | 21:56 | |
| lifeless | https://etherpad.openstack.org/p/tripleo-test-cluster | 21:56 |
| lifeless | the broker is written | 21:56 |
| lifeless | there is a node definition in nodepool and jobs in the experimental queue | 21:56 |
| *** rongze has quit IRC | 21:57 | |
| clarkb | I think it should be mostly better now. Writing up a bug. | 21:57 |
| clarkb | any tests starting after this point in time should pass | 21:57 |
| lifeless | we're within a week or so of spinning up some test environment machines and being able to start exercising things | 21:57 |
| lifeless | that will be a little bad initially as the tripleo-gate instances would be on the old grizzly cloud (tripleo CD cloud is too unstable for infra as yet) | 21:57 |
| lifeless | but we should be able to get up to silent, perhaps even gate with that | 21:58 |
| lifeless | erm | 21:58 |
| lifeless | silent perhaps even check | 21:58 |
| lifeless | though we'll need multiregion, but thats only a few weeks away before the RH region comes online | 21:58 |
| *** sarob has joined #openstack-infra | 21:58 | |
| *** dolphm has quit IRC | 21:59 | |
| devananda | lifeless: have the link to the nodepool bits? | 21:59 |
| lifeless | devananda: git clone https://github.com/openstack-infra/nodepool, IIRC | 22:00 |
| clarkb | notmyname: https://bugs.launchpad.net/openstack-ci/+bug/1257085 | 22:00 |
| lifeless | devananda: Exactly how we map Ironic specific tests to this environment is extremely flexible | 22:00 |
| uvirtbot | Launchpad bug 1257085 in openstack-ci "Swift functional tests fail to configure devstack." [Undecided,New] | 22:00 |
| devananda | lifeless: also, iiuc, all that is unrelated to the discussions i/we had with -infra folks ~6mo ago about how to get nova-bm tested inside of their existing cloud regions | 22:00 |
| lifeless | devananda: we could just use devstack-gate nodes for you, for instance, if we install the broker client there | 22:00 |
| lifeless | devananda: AFAIK all those discussions were predicated on hope. | 22:01 |
| *** dolphm has joined #openstack-infra | 22:01 | |
| lifeless | devananda: which about 4 months ago we burnt to the ground. | 22:01 |
| devananda | lifeless: has that work been abandoned in favor of the separate-cloud-style stuff? | 22:01 |
| devananda | lifeless: what finally killer her? | 22:01 |
| lifeless | devananda: I'm not aware of any other efforts by -infra in this area that are suitable for Ironic functional tests | 22:01 |
| lifeless | nested virt 30m boots | 22:02 |
| lifeless | LXC can't run tgtd | 22:02 |
| devananda | ah | 22:02 |
| lifeless | either is a big issue; in combination boom | 22:02 |
| lifeless | so in principle we don't care where Ironic runs - it can run in the standard -gate image | 22:02 |
| Alex_Gaynor | dhellmann: +1 from me | 22:02 |
| Alex_Gaynor | (that is, +1 to ignoring my -0 :-)) | 22:03 |
| lifeless | problem is that a qemu VM you test with, will take forever to get up to liveness to validate | 22:03 |
| devananda | yes | 22:03 |
| lifeless | of course, we were testing rather bigger immages that the bare minium you could get away with | 22:03 |
| *** sarob has quit IRC | 22:03 | |
| lifeless | even still... | 22:03 |
| devananda | today's devstack-gate is running with 1 layer of nesting | 22:04 |
| *** markmcclain has quit IRC | 22:04 | |
| devananda | but that isn't seeing 30m boots | 22:04 |
| *** mfer has quit IRC | 22:04 | |
| lifeless | it's running cirros | 22:04 |
| devananda | we should, in theory, be able to do ironic functional tests with no more nesting than devstack-gate has today | 22:05 |
| lifeless | not Ubuntu or Fedora | 22:05 |
| devananda | right | 22:05 |
| devananda | sufficient for ironic. not for tripleo | 22:05 |
| lifeless | sure | 22:05 |
| lifeless | so - like I say, I'm not aware of anyone pushing on an effort to do that w/Ironic or Nova BM | 22:05 |
| devananda | gotcha | 22:05 |
| lifeless | I agree that it is in principle doable. | 22:05 |
| devananda | pleia2: were you looking into ^ at one point in time? | 22:06 |
| devananda | lifeless: so the question seems to be, is it worthwhile to do that? | 22:06 |
| devananda | given the progress ya'll are making on separate clouds for tripleo testing | 22:06 |
| pleia2 | openstack on qemu is unusable | 22:07 |
| lifeless | pleia2: devananda is querying just the bm deployment layer | 22:07 |
| devananda | is there a benefit to getting ironic tested in a devstack-gate-like environment, with cirros, and the ssh driver | 22:07 |
| lifeless | pleia2: so booting cirros via pxe | 22:07 |
| *** mrmartin has quit IRC | 22:07 | |
| jeblair | devananda: i think it makes sense if you want ironic to be tested in a gating environment by icehouse2. | 22:08 |
| pleia2 | ah, I see | 22:08 |
| devananda | jeblair: which I do | 22:08 |
| lifeless | there's a much smaller dependency chain to it | 22:08 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Add noqa support for H201 (bare except) https://review.openstack.org/57334 | 22:08 |
| devananda | jeblair: well. voting. smoke check. not gating yet b/c it's not integrated yet. | 22:08 |
| *** sarob has joined #openstack-infra | 22:08 | |
| jeblair | devananda: right; though you are free to gate ironic on such tests. | 22:08 |
| devananda | lifeless: functional testing in the gate pipeline (even if it doesn't include actual physical machines) is required for ironic in a fairly short time frame | 22:09 |
| devananda | jeblair: right | 22:09 |
| jeblair | devananda: (it would be asymmetric, so should be done with care, but anyway, that's a side track) | 22:09 |
| devananda | jeblair: we're 1 patch away from having devstack / tempest tests in our pipeline: https://review.openstack.org/#/c/53917 | 22:10 |
| devananda | and there's a similar patch for tempest's experimental pipe | 22:10 |
| lifeless | so, I think you should get devstack-gate up regardless | 22:10 |
| *** dkranz has quit IRC | 22:10 | |
| jeblair | devananda: i'm excited and optimistic about the tripleo-based testing but would not want too many things to depend on such a large unknown at this point; and i'd like the tripleo work to proceed without the kind of pressure this would add | 22:10 |
| jeblair | lifeless: ++ | 22:10 |
| lifeless | there's a short path to it, and if we can do something better in future cool, but why add risk we don't need to | 22:10 |
| devananda | awesome | 22:10 |
| *** dolphm has quit IRC | 22:11 | |
| *** ArxCruz has joined #openstack-infra | 22:12 | |
| devananda | pleia2: have a few minutes to brain dump where you got to with adding nova-bm to d-g? | 22:13 |
| pleia2 | devananda: sure | 22:13 |
| pleia2 | devananda: here or would you prefer a call? | 22:14 |
| *** ^demon|lunch is now known as ^d | 22:14 | |
| devananda | pleia2: which ever is easier | 22:15 |
| mordred | dhellmann: did we not ever release oslo.messaging 1.2 ? | 22:15 |
| pleia2 | devananda: quick call might be easiest, terms get confusing :) | 22:15 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Use purge option where appropriate. https://review.openstack.org/59553 | 22:15 |
| *** fifieldt has joined #openstack-infra | 22:16 | |
| clarkb | sdague: dhellmann: for gating WSME/Pecan on devstack/tempest jobs. You can now overload the branch-designator to include a unique string for wsme/pecan | 22:16 |
| mgagne | with hpcloud, how can I make use of the API key? (in place of password) | 22:16 |
| clarkb | sdague: dhellmann: that will give you an otherwise identical job to all of the other jobs, but the name will be different so zuul won't run them all in the same gate | 22:16 |
| *** geekinutah has joined #openstack-infra | 22:18 | |
| *** zul has quit IRC | 22:18 | |
| *** zul has joined #openstack-infra | 22:20 | |
| *** pabelanger_ has quit IRC | 22:21 | |
| *** pabelanger has joined #openstack-infra | 22:21 | |
| *** dkliban has quit IRC | 22:25 | |
| *** dizquierdo has quit IRC | 22:27 | |
| *** rwsu has quit IRC | 22:27 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Make job names shorter https://review.openstack.org/59564 | 22:28 |
| jeblair | clarkb: you may be interested in that ^ | 22:30 |
| *** sarob has quit IRC | 22:30 | |
| ruhe_ | jeblair, should this https://review.openstack.org/#/c/57317 be modified to comply with new naming style? ^ | 22:32 |
| sdague | mordred: apparently yelling on twitter about broken openstack things get's johnathan to email you :) | 22:33 |
| mordred | sdague: nice! | 22:33 |
| mordred | sdague: what a great and open process | 22:33 |
| sdague | I put you into that email thread as well | 22:33 |
| mordred | please do | 22:33 |
| *** ruhe_ is now known as ruhe | 22:34 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Gate current clients on stable branches https://review.openstack.org/41931 | 22:35 |
| clarkb | mordred: ^ updated that change to be more like what we are doing with the other jobs | 22:36 |
| clarkb | jeblair: looking | 22:36 |
| *** loq_mac has quit IRC | 22:38 | |
| *** mriedem has quit IRC | 22:39 | |
| clarkb | jeblair: I like that. I will rebase 41931 on it probably | 22:39 |
| *** CaptTofu has quit IRC | 22:40 | |
| *** CaptTofu has joined #openstack-infra | 22:40 | |
| mordred | clarkb: looks sexy | 22:42 |
| *** rwsu has joined #openstack-infra | 22:43 | |
| clarkb | I am going to try and do some reviews now | 22:45 |
| *** MarkAtwood has joined #openstack-infra | 22:45 | |
| clarkb | jeblair: fungi: mordred: once the d-g job name scheme settles we should do a massive job cleanup on the jenkins masters | 22:46 |
| *** _david_ has quit IRC | 22:49 | |
| *** dims has quit IRC | 22:51 | |
| jeblair | clarkb: yes | 22:53 |
| *** rongze has joined #openstack-infra | 22:53 | |
| *** fbo is now known as fbo_away | 22:53 | |
| *** thomasem has quit IRC | 22:53 | |
| *** leifmadsen has joined #openstack-infra | 22:54 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Make job names shorter https://review.openstack.org/59564 | 22:56 |
| *** bpokorny has quit IRC | 22:56 | |
| *** rongze has quit IRC | 22:58 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Make job names shorter https://review.openstack.org/59564 | 22:59 |
| *** leifmadsen has quit IRC | 22:59 | |
| fungi | clarkb: it's way, way, way easier now that jjb has an option for that | 22:59 |
| clarkb | jeblair: I am going to hold off on additional d-g change until we can get your rename in | 23:00 |
| *** flaper87 is now known as flaper87|afk | 23:00 | |
| jeblair | clarkb: k. i think my patch may need a similar puppet stoppage | 23:00 |
| *** denis_makogon has quit IRC | 23:00 | |
| *** denis_makogon has joined #openstack-infra | 23:01 | |
| *** pete5 has quit IRC | 23:02 | |
| clarkb | jeblair: yup | 23:02 |
| clarkb | jeblair: https://review.openstack.org/#/c/54331/9/nodepool/provider_manager.py wouldn't it be better to pass pool in where addPublicIP is called? | 23:03 |
| *** sarob has joined #openstack-infra | 23:04 | |
| jeblair | clarkb: yes | 23:05 |
| clarkb | I am leaving a review | 23:05 |
| *** dolphm has joined #openstack-infra | 23:08 | |
| *** ryanpetrello has quit IRC | 23:08 | |
| *** denis_makogon has quit IRC | 23:09 | |
| *** jcooley_ has joined #openstack-infra | 23:09 | |
| *** esker has quit IRC | 23:09 | |
| clarkb | jeblair: jenkins still isn't happy with the rename change | 23:11 |
| jeblair | clarkb: thx; will fix | 23:12 |
| openstackgerrit | Yathiraj Udupi proposed a change to openstack/requirements: Added a requirement to COIN PULP LP Modeler module https://review.openstack.org/53446 | 23:13 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Make job names shorter https://review.openstack.org/59564 | 23:16 |
| jeblair | clarkb, mordred, fungi: fyi i'm doing some testing with a manually created job called 'dev-nova-pep8' on jenkins01 | 23:18 |
| fungi | okay | 23:18 |
| *** jergerber has quit IRC | 23:18 | |
| *** hashar has quit IRC | 23:18 | |
| fungi | i | 23:19 |
| fungi | er | 23:19 |
| fungi | i'll refrain from deleting it | 23:19 |
| jeblair | fungi: i won't be upset if it is deleted; i know what i'm getting into. :) | 23:19 |
| *** dkliban has joined #openstack-infra | 23:20 | |
| ekarlso | any chance to get this merged now ? https://review.openstack.org/#/c/56927/ | 23:20 |
| clarkb | ekarlso: mordred was looking into why newer jeepyb wasn't getting installed onreview.o.o | 23:21 |
| clarkb | ekarlso: once we hear back on that that things are good we can start approving new project changes | 23:21 |
| clarkb | looks like the version is newer now than what it was before | 23:22 |
| clarkb | mordred: did you fix that? what did you have to do? | 23:22 |
| clarkb | jeblair: fungi: mordred: we can guinea pig with https://review.openstack.org/#/c/56107/1 to see it work on review.o.o | 23:24 |
| fungi | clarkb: checking it out now, but looks like it's probably a safe one to test with | 23:26 |
| clarkb | jeepyb version reported by pip on review.o.o is current. I am guessing I missed what mordred did to fix that in scrollback? | 23:28 |
| fungi | clarkb: i don't think he said what he did | 23:29 |
| fungi | (or else it self-corrected) | 23:29 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add support for installing the web dashboard https://review.openstack.org/59289 | 23:31 |
| *** jcooley_ has quit IRC | 23:32 | |
| *** Ryan_Lane has joined #openstack-infra | 23:33 | |
| *** yamahata_ has quit IRC | 23:34 | |
| *** herndon_ has quit IRC | 23:35 | |
| *** krotscheck has quit IRC | 23:35 | |
| *** FallenPegasus has joined #openstack-infra | 23:36 | |
| *** FallenPegasus has quit IRC | 23:36 | |
| *** Ryan_Lane has quit IRC | 23:37 | |
| *** sandywalsh has quit IRC | 23:37 | |
| *** jcoufal has quit IRC | 23:38 | |
| clarkb | jeblair: https://review.openstack.org/#/c/59564/4/modules/openstack_project/files/jenkins_job_builder/config/python-bitrot-jobs.yaml I don't see the change in projects.yaml to change the branch names to prefix with - | 23:38 |
| clarkb | oh wait, we instantiate the templates in the job group | 23:38 |
| *** yamahata_ has joined #openstack-infra | 23:39 | |
| jeblair | clarkb: all set? | 23:39 |
| *** krotscheck has joined #openstack-infra | 23:39 | |
| clarkb | jeblair: I think so. I gave it a +2 | 23:39 |
| *** marun has quit IRC | 23:40 | |
| *** jerryz has joined #openstack-infra | 23:44 | |
| *** sarob has quit IRC | 23:45 | |
| *** sarob has joined #openstack-infra | 23:46 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a bare precise node to nodepool https://review.openstack.org/59572 | 23:47 |
| clarkb | jeblair: can you review https://review.openstack.org/#/c/56107/ and approve (remove the -2) if it looks good to you? | 23:48 |
| clarkb | now for some hpcloud new region nodepooling /me digs up that change | 23:48 |
| jeblair | clarkb: you babysitting it? | 23:48 |
| clarkb | jeblair: I can | 23:49 |
| jeblair | clarkb: aprvd | 23:49 |
| fungi | i will also babysit that if nedded | 23:49 |
| clarkb | ty | 23:49 |
| fungi | er, needed | 23:49 |
| *** ruhe has quit IRC | 23:50 | |
| clarkb | fungi: "nedded": verb, to be greeted with a "Hey Diddly Ho neighbor" | 23:50 |
| *** emagana has quit IRC | 23:50 | |
| fungi | hidey-ho, neighbor! | 23:50 |
| fungi | [you have been nedded] | 23:51 |
| clarkb | mordred: can you confirm on https://review.openstack.org/#/c/56260/ that we should go ahead and start trying the new region? | 23:51 |
| clarkb | mordred: maybe mark it ready for review | 23:52 |
| *** sandywalsh has joined #openstack-infra | 23:53 | |
| mordred | clarkb: done | 23:53 |
| clarkb | woot, I will approve that once I am done babysitting the new project change | 23:54 |
| fungi | clarkb: jeblair: https://review.openstack.org/#/c/59564/4/modules/openstack_project/files/jenkins_job_builder/config/python-bitrot-jobs.yaml we're ending up with job names like "periodic-nova-python27havana" (threw away a hypnen... intended?) | 23:54 |
| *** bpokorny has joined #openstack-infra | 23:55 | |
| *** rongze has joined #openstack-infra | 23:55 | |
| fungi | (hyphen too. damned fingers) | 23:55 |
| clarkb | fungi: nope, that was my question too. see the end of that file, we set the parameter values there and not in projecs.yaml | 23:56 |
| fungi | so right | 23:56 |
| *** thedodd has quit IRC | 23:57 | |
| *** UtahDave has quit IRC | 23:57 | |
| fungi | branch-name: '-havana' | 23:57 |
| fungi | ignore me | 23:57 |
| fungi | i should hold my questions until the end of the presentation | 23:57 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Update job names https://review.openstack.org/59573 | 23:58 |
| *** CaptTofu has quit IRC | 23:58 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: raise JenkinsJobsException instead of Exception https://review.openstack.org/56397 | 23:58 |
| *** CaptTofu has joined #openstack-infra | 23:58 | |
| jeblair | sdague, mtreinish, jog0: what's the deal with the query bug 1250168 ? it doesn't seem like including a job name in a query is a good idea..? | 23:58 |
| uvirtbot | Launchpad bug 1250168 in nova "gate-tempest-devstack-vm-neutron-large-ops is failing" [High,Fix committed] https://launchpad.net/bugs/1250168 | 23:58 |
| jeblair | s/the query bug/the query for bug/ | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!