Scaladoc not being indexed by Google?

I used the wrong address of scala-user mailing list. Now it should be ok, sorry.

On 27 November 2011 18:09, Grzegorz Kossakowski
wrote:
> Hi,
>
> [cross-posting because topic needs attention from scala committers and
> I hope scala users can help with debugging the issue itself]
>
> I tried to Google something from scaladocs and got really astonished.
> It looks like Google doesn't index scaladoc, at all. See:
>
> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+list+2...
> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+map+2....
>
> Let's try to google for something recently added to trunk:
>
> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+%22tre...
>
> In all three cases all I'm getting is a link to index.html page of
> scaladocs. It looks like all other pages are not visible to Google's
> index. I tried to find out possible reason for this but I don't see
> anything obvious. Robots.txt looks ok, links in scaladoc's menu are
> static (not generated by js).
>
> Does anybody has an idea?

Google had no problem indexing my scaladoc. Mayke sure you have a reachable link that points to your index.html.


On Sun, Nov 27, 2011 at 11:11 AM, Grzegorz Kossakowski <grzegorz.kossakowski@gmail.com> wrote:

I used the wrong address of scala-user mailing list. Now it should be ok, sorry.

On 27 November 2011 18:09, Grzegorz Kossakowski
<grzegorz.kossakowski@gmail.com> wrote:
> Hi,
>
> [cross-posting because topic needs attention from scala committers and
> I hope scala users can help with debugging the issue itself]
>
> I tried to Google something from scaladocs and got really astonished.
> It looks like Google doesn't index scaladoc, at all. See:
>
> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+list+2.9&pbx=1&oq=scaladoc+list+2.9
> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+map+2.9&pbx=1&oq=scaladoc+map+2.9
>
> Let's try to google for something recently added to trunk:
>
> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+%22treeprinters%22&pbx=1&oq=scaladoc+%22treeprinters%22
>
> In all three cases all I'm getting is a link to index.html page of
> scaladocs. It looks like all other pages are not visible to Google's
> index. I tried to find out possible reason for this but I don't see
> anything obvious. Robots.txt looks ok, links in scaladoc's menu are
> static (not generated by js).
>
> Does anybody has an idea?

--
Grzegorz Kossakowski

On close inspection it looks like Google does not index inside the docs. So you are correct, there is a problem in there somewhere.


On Sun, Nov 27, 2011 at 11:15 AM, Aleksey Nikiforov <lexn82@gmail.com> wrote:

Google had no problem indexing my scaladoc. Mayke sure you have a reachable link that points to your index.html.

On 27 November 2011 18:15, Aleksey Nikiforov wrote:
> Google had no problem indexing my scaladoc. Mayke sure you have a reachable
> link that points to your index.html.

It does index index.html but nothing else. That's how it looks like, at least.

On 2011-11-27 18:09, Grzegorz Kossakowski wrote:
> I tried to Google something from scaladocs and got really astonished.
> It looks like Google doesn't index scaladoc, at all. See:
>
> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+list+2...
> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+map+2....

I usually google for "scala list" or "scala map" which leads me straight
to the nightly docs. The links are not as useful as they could be though
because they only lead to the individual pages and not to the frameset.
I recently filed https://issues.scala-lang.org/browse/SI-5212 for that.
I suppose not being able to find API docs for a specific version number
is a related problem. The version number is on the frameset title but
not on the individual pages, so you can't google for the combination of
a single page with a specific API version.

> Let's try to google for something recently added to trunk:
>
> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+%22tre...

Googling for "scala nsc treeprinter" works for me. Googling for
"scaladoc" probably has the same issue as the version number queries
above: The word "scaladoc" does not occur anywhere on those pages, and
links that mention scaladoc will go to the index page instead.

Cheers,
Stefan

On 28 November 2011 21:13, Stefan Zeiger wrote:
> On 2011-11-27 18:09, Grzegorz Kossakowski wrote:
>>
>> I tried to Google something from scaladocs and got really astonished.
>> It looks like Google doesn't index scaladoc, at all. See:
>>
>>
>> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+list+2...
>>
>> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+map+2....
>
> I usually google for "scala list" or "scala map" which leads me straight to
> the nightly docs. The links are not as useful as they could be though
> because they only lead to the individual pages and not to the frameset. I
> recently filed https://issues.scala-lang.org/browse/SI-5212 for that. I
> suppose not being able to find API docs for a specific version number is a
> related problem. The version number is on the frameset title but not on the
> individual pages, so you can't google for the combination of a single page
> with a specific API version.

Oh yes. Those are all good catches. I'll file bug for title bug.

>
>> Let's try to google for something recently added to trunk:
>>
>>
>> https://www.google.com/#sclient=psy-ab&hl=en&source=hp&q=scaladoc+%22tre...
>
> Googling for "scala nsc treeprinter" works for me. Googling for "scaladoc"
> probably has the same issue as the version number queries above: The word
> "scaladoc" does not occur anywhere on those pages, and links that mention
> scaladoc will go to the index page instead.

Yup. Other thing that I wonder about is why Scaladoc have such a low
page rank. Probably another thing to find out.