[PATCH] hiding form pages from search engines (Feature #169)

I guess having something like */new in the robots.txt wouldn't work, would it?

According to http://www.robotstxt.org/robotstxt.html, robots.txt does not support glob. So we can not expect */new works fine.

I think this is a good idea. I'd like to improve on it a little bit though by making the robot_exclusion_tag take options for the content section (e.g. robot_exclusion_tag("NOINDEX,FOLLOW,NOARCHIVE") or robot_exclusion_tag("NOINDEX,NOFOLLOW")). Then we (or plugins) could have more control over the indexing options for each page.

Thoughts?

Eric Davis wrote:

Thoughts?

What about making NOINDEX,FOLLOW,NOARCHIVE the default and calling the method with any collection of (NO)SOMETHING overrides the default for that keyword?

This was my idea. It let us have more control of what the actual content is in case the meta tag allows other values later.

1  def robot_exclusion_tag(content="NOINDEX,FOLLOW,NOARCHIVE")
2    "<meta name='ROBOTS' content=#{content} />"    
3  end

Eric Davis wrote:

This was my idea. It let us have more control of what the actual content is in case the meta tag allows other values later.

No, I meant having NOINDEX,FOLLOW,NOARCHIVE be the default, and if you call it with INDEX to get INDEX,FOLLOW,NOARCHIVE. I just realized that's overengineering it though, I like your proposal :-)

Looks good to me. I'll merge it by the time I'm around a more stable connection if you haven't done so until then.

I still like Felix' idea of having defaults and being able to gradually overwrite them. This could be done like this:

 1# Add a HTML meta tag to control robots (web spiders)
 2#
 3# @param [optional, String] changed content of the ROBOTS tag.
 4#   defaults to no index, follow, and no archive
 5def robot_exclusion_tag(content="")
 6  default_content = { "INDEX" => "NO",
 7                      "FOLLOW" => "",
 8                      "ARCHIVE" => "NO" }
 9
10  args = content.upcase.split(",").inject({}) do |args, arg|
11    value = arg.gsub(/^(NO)/, "")
12    args[value] = $1 || "" 
13    args
14  end
15  default_content.merge(args).collect{ |k, v| v+k }.join(",")
16end

Holger Just wrote:

I still like Felix' idea of having defaults and being able to gradually overwrite them.

Just seems like a lot of code to me that might not be used that often.

Eric Davis wrote:

Just seems like a lot of code to me that might not be used that often.

That's what I meant with "don't overengineer it" ;-) I think the simple version is fine, avoiding to have to write it all out for those few times you need other params is not worth it.