Jamit Software Forum
Welcome, Guest. Please login or register.
July 29, 2021, 03:00:37 am

Login with username, password and session length
Search:     Advanced search
May 5th, 2010 - Jamit Software Launches the Market!
3,080 Posts in 791 Topics by 1,787,675 Members
Latest Member: bvbwziwmxfd
* Home Help Search Login Register
+  Jamit Software Forum
|-+  Jamit Job Board Customers
| |-+  User-to-User Support
| | |-+  robots.txt file, best advice ....
« previous next »
Pages: [1] Print
Author Topic: robots.txt file, best advice ....  (Read 5969 times)
darkstaruk
Jammers
Jr. Member
*
Posts: 13


« on: September 25, 2009, 02:13:01 pm »

hi,  

any advice on how to construct a decent robots file? i think i have done it..... i have done the following, not sure
if its correct or not?  e.g. should i disallow all these files???

User-agent: *
Sitemap: http://www.mysite.com/job-xml-sitemap.php
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /cache/
Disallow: /cron /
Disallow: /docs/
Disallow: /employers/
Disallow: /fckededitor/
Disallow: /include/
Disallow: /lang/
Disallow: /mail/
Disallow: /menu/
Disallow: /myjobs/
Disallow: /offline/
Disallow: /overlib/
Disallow: /payment/
Disallow: /scw/
Disallow: /upload_files/
Disallow: /wz_dragdrop/
Disallow: /offline/
Disallow: /secure/
Disallow: /rss.php/
Disallow: /su.php/
Disallow: /p.php/
Disallow: /lang_image.php/
Disallow: /list_iframe.php/
Disallow: /jb-xml-pickup.php/
Disallow: /jb-get-xml.php/
Disallow: /error_log/
Disallow: /email_job_window.php/
Disallow: /db.php/
Disallow: /appurl_iframe.php/
Disallow: /apply_iframe.php/


many thanks
  
« Last Edit: November 29, 2009, 01:40:55 pm by darkstaruk » Logged
rutulo
Jammers
Sr. Member
*
Posts: 40


« Reply #1 on: November 18, 2009, 11:15:18 am »

I'm interessed at this... can I know if this file is correct?
thank's
Logged
darkstaruk
Jammers
Jr. Member
*
Posts: 13


« Reply #2 on: November 29, 2009, 01:40:25 pm »

can anyone advice on what to put in the robots file???

leave it totally blank???  or Huh?
Logged
jaffri
Jammers
Sr. Member
*
Posts: 51


« Reply #3 on: February 26, 2010, 01:05:01 pm »

The above list "looks" quite appropriate. Can someone else with knowledge of this, please comment.
Logged
darkstaruk
Jammers
Jr. Member
*
Posts: 13


« Reply #4 on: December 10, 2010, 07:55:03 pm »

i would still like to know whats the best robots file for jamit please  Roll Eyes Huh :'(
Logged
Peter
Administrator
Hero Member
*****
Posts: 248



« Reply #5 on: December 11, 2010, 01:51:57 am »

robots.txt ..... is only good for honest robots (spiders). It tells them which pages they should NOT index. They still will access that page (document, image etc.), but if they are honest, they will not index it.

Only a few robots are honest. Those probably include only Google, Yahoo, MSN, but don't take my word for it.

If you have a secret document, image, folder etc. and you don't want anyone to know, DO NOT mention it in your robots.txt !!!!

Incorrectly written robots.txt file is basically the giveaway -- look at my secret files! Bad robots look there first and then they access all those files listed.

Here is an example of a robots.txt (although it may not be applicable to Job Board):

Code:
User-Agent: *

Disallow: /*.ico$
Disallow: /*.jpg$
Disallow: /*.gif$
Disallow: /*.png$
Disallow: /*.mp3$
Disallow: /*.flv$
Disallow: /*.swf$
Disallow: /*.css$
Disallow: /*.js$

Sitemap: /sitemap.xml
« Last Edit: December 11, 2010, 01:54:03 am by Peter » Logged

SECURE your site BEFORE you wish you had! Use plugins by COLOSSAL MIND!
zorab
Guest
« Reply #6 on: January 04, 2011, 09:12:42 pm »

Hi there.

It looks for me that you are disallowing everything darkstaruk
Whats left to allow anyway.
I think its better to say what is ok to allow or disallow.
Logged
Pages: [1] Print 
« previous next »
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!
Page created in 0.221 seconds with 17 queries.