Thursday, 2015-07-09

*** puranamr has joined #openstack-community00:07
*** Piet has joined #openstack-community00:14
*** puranamr has quit IRC00:25
*** puranamr has joined #openstack-community00:28
*** Piet has quit IRC00:51
*** thickskin has left #openstack-community00:53
*** Piet has joined #openstack-community00:57
*** xiangxinyong has joined #openstack-community01:49
*** puranamr has quit IRC02:09
*** xiangxinyong1 has joined #openstack-community02:18
*** xiangxinyong has quit IRC02:22
*** hakimo has joined #openstack-community02:52
*** xiangxinyong1 is now known as xiangxinyong02:52
*** hakimo_ has quit IRC02:54
*** rbrooker has quit IRC02:56
*** puranamr has joined #openstack-community03:08
*** puranamr has quit IRC03:08
*** julim has quit IRC03:36
*** julim has joined #openstack-community03:37
*** julim has quit IRC03:37
*** Marga_ has joined #openstack-community04:02
*** Marga__ has joined #openstack-community04:02
*** Marga_ has quit IRC04:07
*** xiangxinyong has quit IRC04:28
*** ozstacker_ has joined #openstack-community04:31
*** ozstacker has quit IRC04:35
*** neeti has joined #openstack-community05:22
*** ig0r_ has joined #openstack-community05:52
*** ig0r__ has quit IRC05:55
*** xiangxinyong has joined #openstack-community06:04
*** jtomasek has joined #openstack-community06:17
*** hakimo has quit IRC06:40
*** SamYaple has joined #openstack-community07:07
SamYapleask.openstack.org needs some love. lots of spammers07:08
SamYapleive been cleaning it up, but block_spammers is deadlocking the DB due to number of entries I believe07:08
SamYaplereed ^^07:08
*** Piet has quit IRC07:10
*** tobe has joined #openstack-community07:18
SamYapleI cleaned up ~4000 spam posts going back about 12 hours from now07:36
*** dizquierdo has joined #openstack-community07:37
*** dizquierdo has quit IRC07:38
*** dizquierdo has joined #openstack-community07:38
*** dpose has joined #openstack-community07:41
*** dizquierdo_ has joined #openstack-community08:43
*** dizquierdo has quit IRC08:47
*** dizquierdo_ has quit IRC08:59
*** r-mibu has joined #openstack-community09:00
*** r-mibu has left #openstack-community09:00
*** coolsvap|away is now known as coolsvap09:15
*** dpose has quit IRC09:35
*** dpose has joined #openstack-community09:36
*** dizquierdo has joined #openstack-community09:37
*** coolsvap is now known as coolsvap|away09:48
*** dpose has quit IRC10:06
*** dpose has joined #openstack-community10:06
*** coolsvap|away is now known as coolsvap10:11
*** dpose has quit IRC10:14
*** coolsvap is now known as coolsvap|away10:25
*** xiangxinyong has quit IRC10:39
*** xiangxinyong has joined #openstack-community11:20
*** coolsvap|away is now known as coolsvap11:22
*** tobe has quit IRC11:48
*** xiangxinyong has quit IRC12:06
*** annegentle has joined #openstack-community12:15
*** sgordon has joined #openstack-community12:23
sgordonwell then12:23
sgordonlooks like there might be some fun clean up to do on ask.o.o12:24
SamYapleyea sgordon i couldnt keep up12:24
SamYaplegot stuff to do12:24
SamYaple07:08 < SamYaple> ask.openstack.org needs some love. lots of spammers12:24
SamYaple07:09 < SamYaple> ive been cleaning it up, but block_spammers is deadlocking the DB due to number of entries I12:24
SamYaple                  believe12:24
SamYaple07:09 < SamYaple> reed ^^12:24
SamYaple07:37 < SamYaple> I cleaned up ~4000 spam posts going back about 12 hours from now12:24
sgordonlemme see12:24
SamYaple12:24 < sgordon> well then12:24
sgordoni think i can block user and del all content12:24
SamYaplesgordon: you can, but there are lots of users12:25
sgordonfun12:25
SamYaplei cleaned up 4000 posts, about 200 hundred users12:25
SamYaplelooks like there is that much work to do now as well12:25
sgordonany idea why they are getting straight through instead of going into the mod queue?12:25
SamYaplenope12:26
SamYaplecleaning up allowed me to cause several database deadlocks though :)12:26
sgordonfifieldt, do you know if anyone can help us out temporarily by blocking new user registration or something ^12:28
SamYaplereed should be able too12:28
*** annegentle has quit IRC12:38
*** julim has joined #openstack-community12:44
*** rbowen has quit IRC12:45
*** pserebryakov has joined #openstack-community12:47
*** neeti has quit IRC12:49
fifieldthi12:53
SamYaplehello12:53
fifieldtso, sgordon, SamYaple12:53
fifieldtreed used to take care of ask's spam thingy12:53
fifieldtat the moment, fungi might have access12:53
SamYaplewhat happened to reed?12:54
fifieldthe left12:54
SamYapleI was unaware12:54
fifieldtlast time this happened an IP address block was effective12:54
SamYaplelots of ips12:54
fifieldtyes, we blocked a range12:54
SamYaplelots of ranges12:55
fifieldtwe basically blocked a major ISP in India12:55
fifieldtfor 3 days12:55
fifieldtafter which the attack stopped12:55
fifieldtthe spam tools in ask are not effective to deal with massive attacks12:55
SamYaplethis is korea and russia12:55
fifieldtyou guys in the admin queue blocking folks?12:56
SamYaplefifieldt: yea but i cant stop new registrations12:56
SamYapleive been fighting it for about 10 hours waiting for someone with more server control12:56
SamYaplegots stuff to do12:56
fifieldtfungi would be it, as far as I know12:56
fifieldtI'll see if I can raise him12:57
SamYapleto be fair today is my day off and i was also watching the termintator movies, so no real loss12:57
sgordonfifieldt, trying to but if you look at the new users queue they have 100+ accounts12:57
sgordonand the biggest hammer we have is to go through each one and block+delete12:57
fifieldtyup, askbot is totally ineffective to deal with this kind of attack12:58
fifieldtlet;s see if we can work out some ip ranges12:58
fifieldthttps://etherpad.openstack.org/p/ask-spam-ips12:58
SamYaplei dont think i have access to a users queue, i just have moderation queue12:58
sgordonSamYaple, i dont either12:59
sgordonSamYaple, just looking at https://ask.openstack.org/en/users/12:59
SamYapleah12:59
*** Marga__ has quit IRC13:00
fifieldtfor reference, the previous spam attack was never properly cleaned up13:01
SamYaplei know :)13:01
SamYaplelooks like alot of korea13:02
fifieldtmy "Block Spammers" button in the moderation queue isn't even working13:02
SamYaplefifieldt: it is, its just deadlocking the database if you do more than one user13:03
SamYapledont do that13:03
fifieldtgreat software :)13:03
sgordonyou have to go to their user page13:03
sgordonand click block and delete all content13:03
sgordonit's da best13:03
SamYaplesgordon: thats what block spammers does in the mod queue13:04
sgordonahk13:04
sgordonwell same as for fifieldt that button doesnt actually work for me13:04
SamYapleone user at a time13:04
SamYaplefigured that out within the first 20 deadlocks :)13:04
sgordoneven with 113:04
sgordonit just does nada13:05
*** rbowen has joined #openstack-community13:05
SamYapleits working for me13:05
*** Marga_ has joined #openstack-community13:05
sgordonyay i am down to NerdSkills1113:06
sgordon;p13:06
SamYaplei remember nerdskills0113:06
SamYaplei just cleaned up nerdskills50 :/13:06
*** Marga_ has quit IRC13:07
*** Marga_ has joined #openstack-community13:07
sgordonwoot13:10
sgordonthink we got them all13:10
sgordonfor now13:10
sgordon...13:10
SamYaplenope13:10
SamYaplesgordon: you in the mod queue?13:10
sgordonim looking on the ask.openstack.org front page13:10
sgordonanything that is stuck in the mod queue at this point wasnt actually posted13:11
sgordon(publicly that is)13:11
SamYaplefair point13:11
sgordoni would love to know how all the rest of it didnt end up in the mod queue though13:11
sgordoni would have thought a brand new user immediately posting 10+ questions would be a gimme13:11
SamYaplewe overloaded the mod queue maybe?13:12
SamYapleafter 1000 messages it skips the queue13:12
SamYapleim guessing13:12
sgordonmmm some of the stuff in the queue13:12
sgordonis for users i already blocked and deleted all content for13:12
sgordonso i guess it doesnt clear the entry in the mod queue when you do it from the other button O.o13:12
*** rbrooker has joined #openstack-community13:13
SamYaplesgordon: thanks!13:14
* SamYaple grumbles13:14
SamYaple:)13:14
sgordonunfortunately im sure they will be back13:14
sgordonthe most recent posts were from less than an hour ago13:14
SamYapleand in greater numbers13:14
sgordonand as i say that13:14
sgordonbrand new ones13:14
SamYaplesgordon: this has been going on for about 20 hours13:15
SamYaplei got here 10 hour in13:15
sgordonyeah13:15
fifieldtok13:15
fifieldtI got the admin interface13:15
fifieldtlearning it now13:15
SamYaplemod queue empty again13:16
*** puranamr has joined #openstack-community13:16
fifieldtthis admin interface is rather .. limited13:17
SamYaplefifieldt: it was mostly south korea, a few scattered ips and then a /2613:17
fifieldtwould you recommend a block?13:17
SamYapleon the ips listed from korea and digital ocena, yes13:17
SamYapledigital ocean ones not long though, maybe a few days13:18
SamYaplethose ips cycle13:18
fifieldtaye13:18
fifieldtthough, theory: most of our users probably don't access ask via a proxy hosted on a cloud provider13:18
SamYaplefair point13:19
SamYaplebut then again, I do from RAX for alot of stuff13:19
SamYapleor userbase is the technical kind13:19
fifieldtthe bad news is the admin interface is sadly limited13:20
SamYapleyea :/13:20
*** jehb has joined #openstack-community13:21
fifieldtit's the standard django interface13:22
fifieldtwhich means it's just the database13:22
fifieldtthere's no ability to alter the moderation or subscription methods that I can see13:22
fifieldtoh, wait13:23
fifieldtthere is _another_ admin interfac13:23
fifieldtdo the spam posts have links in them>?13:23
SamYaplenot always13:24
SamYaplesomething about bamwar5.com or something, but it was unicode and not a link13:24
fifieldtok, let me see if there's a string block13:25
SamYaplethat would be most helpful. these spam attacks have had very consistent strings13:25
SamYaplean interface that would let me set the string to block would also be nice13:26
fifieldtyup, looks like there is no such feature here13:26
SamYaple:D13:27
fifieldtLOL13:28
fifieldt" Enable Akismet spam detection(keys below are required) "13:28
fifieldtso13:28
fifieldttake a guess as to whether that is on or off13:28
SamYaplei guess on13:28
fifieldtit was off13:28
fifieldtI have turned it on13:28
SamYaplekinda figured :P13:28
fifieldtand now wonder what it does :)13:28
fifieldt:D13:28
fifieldtok, there's also the potential to turn on a CAPTCHA13:29
fifieldtis that worth doing?13:29
SamYaplei suppose so, but man do i hate capthca13:30
fifieldtoh! I just found how to delete tags13:30
sgordoni actually think a captcha is the right long term solution13:30
sgordonat the very least for signups13:30
fifieldtI think this is for posting :|13:31
sgordonbummer13:31
SamYaplesignups require external auth anyway13:31
SamYapletehre isnt local ask.openstack.org signins anymore13:31
SamYaplenot for a long time13:31
sgordonyeah im saying it would need to be an additional step13:31
SamYaplethe other auths handle that13:31
sgordonyeah well that's clearly working really well13:32
sgordonvery loose definition of 'handle' and all :p13:32
SamYaplecaptcha is really easy to beat nowadays13:32
sgordondo we know which auth provider these came from?13:32
fifieldtok, so how sure are we we got all the spam?13:32
sgordonthat really depends on the captcha13:32
SamYaplepay a guy to crack 30-40 a minute13:32
fifieldtbecause if I delete the tags, we lose an easy way to find it13:32
sgordonfifieldt, right no wit looks clean but let me go back a few pages13:33
SamYaplefifieldt: yea its clean13:33
sgordonyeah13:33
sgordonnada13:33
fifieldtyou guys are legends13:33
fifieldtok, manually deleting tags, 100 at a time :(13:33
SamYaplelol13:33
sgordonit is almost like ask was designed by spammers for spammers13:33
sgordonlol13:33
fifieldtit's a 7-click, two-refresh process13:33
fifieldthaha, it certainl;y feels that way13:34
SamYaplegood luck mr fifield. its been an honor and a pleasure13:34
SamYapleill tell the team where you are13:34
fifieldtlol, thank you guys very, very much13:34
SamYapleyoull be there a while13:34
*** lucas__ has joined #openstack-community13:39
*** puranamr has quit IRC13:45
fifieldtthere's also an option in the admin panel to ask moderators to OK tag creation13:45
SamYaplesounds useful13:46
fifieldtthings that are "prevention" are useful :)13:47
fifieldtbut I wonder what it actually does13:47
fifieldtmaybe it just puts more entries in the moderation queue :P13:47
fifieldtwant to try>?13:47
SamYaplesure13:48
fifieldthmm, can't turn it on and also keep tags required13:51
*** rbrooker has quit IRC13:51
SamYapleit doesnt want you to make it better!13:53
SamYapleits happy just the way it is!13:53
fifieldtwell, yay13:55
fifieldtno ability to sort tags13:55
fifieldtmeaning I do have to go through all 70 pages of them and pick out the bad ones13:55
fifieldtFFFUUUUUU13:55
SamYapleshare the load somehow?13:56
fifieldtit's database query time13:56
SamYapleugh13:56
*** pserebryakov has quit IRC14:00
fifieldtsite's dead14:02
SamYapledammit Tom! youre worse than that dinosaur supervisor14:03
SamYaplewe said FIX not break14:03
SamYapleugh14:03
*** dpose has joined #openstack-community14:04
*** annegentle has joined #openstack-community14:12
*** lucas__ has quit IRC14:27
*** lucas__ has joined #openstack-community14:28
*** macjack has quit IRC14:30
fifieldtok, so we're going to experiment with cloudflare also14:53
fifieldtsince in theory it can CAPTCHA some of the bots14:53
SamYapleok14:54
*** agireud has joined #openstack-community14:57
*** annegentle has quit IRC14:58
reedSamYaple, sgordon, fifieldt: I have enabled "pre-moderation" queue, it  prevents publishing before moderator's decision.15:08
reedon ask.o.o15:08
reedfifieldt, you don't have admin rights on ask? let me check15:08
fifieldtI do now15:08
reedoh, sorry, I wasn't aware15:09
*** coolsvap is now known as coolsvap|away15:09
SamYaplethanks reed that should help in the future15:09
reedon that note, the site needs help to migrate to newer version and establish a process to consume upstream15:11
reedsgordon, do you know who manages askbot on Fedora?15:12
reedwith mrmartin we threw around ideas to fork askbot but we'd need help to maintain it15:13
*** puranamr has joined #openstack-community15:26
*** puranamr has quit IRC15:27
*** puranamr has joined #openstack-community15:28
*** dizquierdo has quit IRC15:32
sgordonreed, i dont know any more than is listed on the wiki15:37
sgordonhttps://fedoraproject.org/wiki/Askbot15:37
sgordoni havent been active on the fedora ask instance in quite some time15:37
reedsgordon, thanks15:38
*** agireud has quit IRC15:40
*** agireud has joined #openstack-community15:43
*** jcoufal has quit IRC15:43
SamYapleive asked larsks reed, he said #fedora-admin is a channel that should have your answer15:46
*** electrocucaracha has joined #openstack-community15:51
reedSamYaple, on freenode, right? /me joins15:53
sgordonyeah15:53
SamYaplereed: yup15:54
*** coolsvap|away is now known as coolsvap15:54
*** lucas__ has quit IRC16:05
*** dpose has quit IRC16:08
*** puiterwijk has joined #openstack-community16:11
reedo/ puiterwijk16:14
puiterwijkhi all16:14
reedfolks, puiterwijk is the fedora packager of askbot16:14
puiterwijk(and Fedora sysadmin responsible for maintaining ask.fedoraproject.org)16:14
reedgreat :)16:15
SamYaplehello puiterwijk!16:15
*** Marga_ has quit IRC16:17
reedpuiterwijk, mrmartin is the person helping openstack on askbot side16:17
reedhere is the latest puppet work done by mrmartin https://review.openstack.org/#/c/172880/16:17
reed(he's not online now, usually he joins later in the evening, CEST time)16:18
reedor CEDT, better :)16:18
puiterwijkreed: checking it out, thanks!16:18
puiterwijkJust curious, what's CEDT?  you mean Central European Summer Time?16:19
reedyeah, daylight saving16:19
puiterwijkhah, unfortunate that I can only login with an Ubuntu account :)16:24
puiterwijkreed: okay, the puppet stuff looks fine to me. So what can I help with on Askbot?16:26
reedpuiterwijk, we'll need to discuss with mrmartin but his idea is to understand better if we can keep a sort of fork16:27
puiterwijkah okay16:28
reedthe fork would just be a collection of upstream commits that we have tested and can 'bless' as stable16:30
puiterwijkwell, would it be an idea to offer upstream some help with gating heir commits?16:31
puiterwijkThat way, everyone would be able to gain from it16:31
*** electrocucaracha has quit IRC16:34
*** annegentle has joined #openstack-community16:35
*** mgarza_ has joined #openstack-community16:35
reedpuiterwijk, right, that would mean setting up gating jobs and such...16:35
reedthat's where help would come in handy16:35
*** coolsvap is now known as coolsvap|away16:36
puiterwijkwell, we can always offer to setup for example travis, which can report back to github (I think that's the main repo, not?). As long as the one merging the commits is forced to at least look at testing results, it's at least an improvement16:36
reedpuiterwijk, the openstack community has jenkins already available, and the idea of a fork is exactly to use the system we already have16:39
reedI think we need to talk to marton about this, i'll send an email so you both know each other and can talk in convenient CEDT/CEST time16:39
puiterwijkright, but I'm doubting whether forking is the best option, or whether we could best just help the current maintainer getting a more sensible merge/release scheme. (just thinking about it, as that would sound ideal to me)16:40
puiterwijkbut sure, if you send the email, I can discuss with your contact about this16:40
*** electrocucaracha has joined #openstack-community16:43
*** lucas__ has joined #openstack-community16:44
*** lucas___ has joined #openstack-community16:46
*** annegentle has quit IRC16:46
*** lucas___ has quit IRC16:47
*** lucas___ has joined #openstack-community16:47
*** lucas__ has quit IRC16:48
*** annegentle has joined #openstack-community16:50
*** lucas___ has quit IRC16:51
*** lucas__ has joined #openstack-community16:57
*** lucas__ has quit IRC17:02
*** fungi has joined #openstack-community17:06
*** mattgriffin has joined #openstack-community17:06
*** coolsvap|away is now known as coolsvap17:08
*** rbrooker has joined #openstack-community17:11
*** lucas__ has joined #openstack-community17:16
*** annegentle has quit IRC17:16
*** Marga_ has joined #openstack-community17:27
*** Marga_ has quit IRC17:30
*** Marga_ has joined #openstack-community17:30
*** lucas__ has quit IRC17:32
*** annegentle has joined #openstack-community17:41
*** lucas__ has joined #openstack-community17:44
*** lucas__ has quit IRC17:51
*** lucas__ has joined #openstack-community17:54
*** agireud has quit IRC17:59
*** annegentle has quit IRC18:00
*** agireud has joined #openstack-community18:10
*** puranamr has quit IRC18:17
*** puranamr has joined #openstack-community18:19
*** annegentle has joined #openstack-community18:19
*** annegentle has quit IRC18:20
*** Marga_ has quit IRC18:24
*** rbrooker has quit IRC18:34
*** mattgrif_ has joined #openstack-community18:36
*** lucas__ has quit IRC18:40
*** mattgriffin has quit IRC18:40
*** mattgrif_ has quit IRC18:50
*** lucas__ has joined #openstack-community18:51
*** puranamr has quit IRC18:52
*** mattgriffin has joined #openstack-community18:53
*** jcoufal has joined #openstack-community19:00
*** coolsvap is now known as coolsvap|away19:04
*** Marga_ has joined #openstack-community19:05
*** puranamr has joined #openstack-community19:11
*** lucas__ has quit IRC19:15
*** lucas__ has joined #openstack-community19:16
*** rbrooker has joined #openstack-community19:19
*** jtomasek has quit IRC19:21
*** Marga_ has quit IRC19:38
*** Marga_ has joined #openstack-community19:39
*** marcoceppi has quit IRC19:39
*** electrocucaracha has quit IRC19:41
*** lucas__ has quit IRC19:43
*** marcoceppi has joined #openstack-community19:49
*** lucas___ has joined #openstack-community19:49
*** lucas___ has quit IRC19:50
*** mattgriffin has quit IRC19:50
*** lucas___ has joined #openstack-community19:51
*** mattgriffin has joined #openstack-community19:52
*** puranamr has quit IRC19:53
*** marcoceppi has quit IRC19:53
*** puranamr has joined #openstack-community19:55
*** lucas___ has quit IRC20:00
*** annegentle has joined #openstack-community20:04
*** puranamr has quit IRC20:05
*** puranamr has joined #openstack-community20:06
*** packet has joined #openstack-community20:16
*** marcoceppi has joined #openstack-community20:18
*** marcoceppi has quit IRC20:18
*** marcoceppi has joined #openstack-community20:18
*** puranamr has quit IRC20:20
*** jcoufal has quit IRC20:26
*** mattgriffin has quit IRC20:27
*** puranamr has joined #openstack-community20:33
*** electrocucaracha has joined #openstack-community20:42
*** lucas__ has joined #openstack-community20:55
*** annegentle has quit IRC20:56
*** julim has quit IRC20:57
*** electrocucaracha has quit IRC20:57
*** packet has quit IRC21:01
*** lucas__ has quit IRC21:03
*** electrocucaracha has joined #openstack-community21:03
*** lucas__ has joined #openstack-community21:10
*** marcoceppi has quit IRC21:11
*** lucas___ has joined #openstack-community21:11
*** lucas__ has quit IRC21:14
*** lucas___ has quit IRC21:15
*** lucas__ has joined #openstack-community21:18
*** marcoceppi has joined #openstack-community21:21
*** lucas__ has quit IRC21:22
*** lucas__ has joined #openstack-community21:30
*** lucas__ has quit IRC21:35
*** agireud has quit IRC21:41
*** coolsvap|away is now known as coolsvap22:05
*** Marga_ has quit IRC22:19
*** coolsvap is now known as coolsvap|afk23:14
*** mgarza_ has quit IRC23:37
*** openstack has joined #openstack-community23:40

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!