Another thing to consider is that some sites make users log in to view certain content or use a form to find content. Logins and forms are bad. The crawlers do not know how to log in to your site or fill out a form in order to view your content. The only content to hide behind a login and robots.txt file is very confidential content you never want indexed.
2. Make your URLs understandable. This means creating URLs like www.colewiebe.com/seo-web-design.html instead of www.colewiebe.com/articles/2007-06-14-article27.html. The first URL is “user friendly”, and it is also “search engine friendly”. The search engines will read more meaning and relevance into the first URL than the second one.
Also, when using the first URL in a blog signature or email, it is clear where the link is going, so click-throughs will increase.
3. Structure your site carefully. Make your site as ‘flat’ as possible. Strive for the magic 3 clicks to whatever content your user is looking for.
4. Design your site for your users, not the search engines . As much as possible treat search engines as if they were human beings manually scouring your site for content to index. If it is easy for humans to find what they are looking for on your site, logic dictates that the spiders will also find what they are looking for and index it.
5. It has often been said that content is king. The search engines job is to index “relevant” search data. When the search engine starts producing “irrelevant” results web users will quicly move onto another engine. Therefore, poor content, poor ranking.
You’re after the triple win. When you develop amazing content the search engines win because they can provide first rate content on the first page of their search results. Other webmasters win when they become recognized as a major resource in their niche by linking to a site like yours. You win by having qualified traffic that builds your business.