extradoc, vscaladoc2 collaboration ?

Hi David,

> Right, it seems, we work in concurrent thread on similar subject. When
> you announced the generation of single json, I did the same (privatly)
> few days before. And (Mea culpa) I didn't notify you nor follow
> evolution of Extradoc (wrongly, I thought it was a demo project to
> show how to generate other output for scaladoc).

Well, I didn't quite knew what it would become back then, so it pretty
much was a demo project. The idea was to do something with the Scala
compiler and getting the docs out in JSON format seemed useful enough.
Then I added a simple JavaScript-based browser for the generated model
(what is now the Model tab), then the navigation bar, some nicer web
design and finally a scaladoc-like presentation (the Page tab).

> Perhaps, we could discuss and collabore (if possible) ?

I just took a closer look at your projects (but I haven't run them yet)
to see where we both stand. Let's examine the different parts:

* JSON format: We're both using a format which is based on the scaladoc2
doc model and a tool which uses the scaladoc2 infrastructure up to and
including the doc model. VScaladoc2 creates lots of individual files
down to the method level.

Extradoc has two modes here: One creates a single file for a
documentation site. Structural redundancies are removed as much as
possible (e.g. inherited methods are not represented by separate
entities but instead their classes link directly to the original
definition; VScaladoc2 seems to do the same) but little is done at the
textual level (except for simple things like omitting definitions with
empty array or string values which VScaladoc2 does not do). It is
expected that such a scaladoc.json file will be gzipped (thus removing
most of the remaining redundancy) and then used as an interchange format
(e.g. for a GUI-based viewer, an IDE, or for merging multiple doc sites
into one).

The other mode (json-multi, also generated along with some static files
in explorer more) is intended for tools like Extradoc Explorer which
examine a single template at a time and can't be expected to download a
complete scaladoc.json for that purpose. It currently generates one file
per doc template (just like scaladoc) plus a separate index
(global.json). In order to show a scaladoc-like presentation for a
template, you will need at most the global index plus the pages for the
template in question plus all templates in its linearization. Entities
will be shared across the linearization but not outside of it, so there
can be some duplicates. In the future, I'd like to merge smaller related
templates (like sub-packages or class hierarchies) if they are too small
for separate files. Having too many small files wastes space for local
doc sites (fs block size) and on-line sites (HTTP request overhead; gzip
compression). The json-multi format also has some further size
optimizations like merged flags (e.g.
"isPublic":true,"isDef":true,"isUseCase":true in json becomes "is":"udU"
in json-multi). The extradoc.js script contains code to undo these
during post-processing. All in all, json-multi is probably the part of
Extradoc on which I spent most of the time.

* Presentation: Is your web site already online? I had a look at the
code and it seems this is where we're really doing different things.
VScaladoc2 has a Lift-based client-server app for generating a HTML
presentation whereas Extradoc Explorer is completely browser-based. The
more ambitious jdocs-like features of VScaladoc2 could not be achieved
in such a way but it can be used locally just like a standard Scaladoc
site. It makes extensive use of jQuery and bits of jQuery UI.

The JSON files are loaded on demand via AJAX and all HTML is generated
from them directly in the browser. The jQuery History plug-in is used to
get a navigation system which resembles are normal web site or a local
scaladoc site consisting of multiple HTML pages. All entities and views
are represented in the hash tag of the URI, so you can bookmark them
individually and use the browser's history.

* Demo project: I don't have one, so I'll try to build yours with
Extradoc. My standard test case is the Scala library because it's rather
large and thus allows me to test performance and memory requirements.

I think the most obvious part for collaboration at this point would be
the JSON format and the tool that generates it. Maybe Extradoc could be
used to generate the data for your web app. I think its JSON generator
is quite mature by now and does lots of optimizations but the code needs
to be cleaned up. It's still rather hackish. What do you think about
Extradoc's single-file and multi-file JSON output? Would they be
suitable for VScaladoc2 or is it better served by its
one-file-per-method format?

> PS: Your wed design is highly better than mine.

Well, I took some design cues from Apple :)

-sz

Hi Stefan,

thank you for (detailed) response.

We started with same idea, one of my first goal was to allow pure web
client (GWT based) as you do, and ease IDE/Editor to retrieve api.
As I need to create a web site (to list api, to list skin, to
consolidate api ref, to search ...) I switch to create a web site and
use it to make the first look and feel (less techno to learn).

I agree about using the same json format (if possible).
I choose to create multi-files, including to use one file for methods
with same name. because I would :
* avoid reader tool (Editor) to load an parse full doc to retrieve
info about method
* inherited method/fields could refer method like non-inherited (
doesn't need to check if api is included or a link)

I admit I didn't take care of optimisation (disk space, ram,...). I
have a to-do list of optimisation (several places for cache, creation
of index on server side,....)

I hope to have a usable online site later next week (I'll be off-line
few days), with a full workable chain. After this step I'll take a
deeper look into extradoc.

Thanks for information

/davidB

On Fri, Oct 29, 2010 at 23:27, Stefan Zeiger wrote:
> Hi David,
>
>> Right, it seems, we work in concurrent thread on similar subject. When
>> you announced the generation of single json, I did the same (privatly)
>> few days before. And (Mea culpa) I didn't notify you nor follow
>> evolution of Extradoc (wrongly, I thought it was a demo project to
>> show how to generate other output for scaladoc).
>
> Well, I didn't quite knew what it would become back then, so it pretty much
> was a demo project. The idea was to do something with the Scala compiler and
> getting the docs out in JSON format seemed useful enough. Then I added a
> simple JavaScript-based browser for the generated model (what is now the
> Model tab), then the navigation bar, some nicer web design and finally a
> scaladoc-like presentation (the Page tab).
>
>> Perhaps, we could discuss and collabore (if possible) ?
>
> I just took a closer look at your projects (but I haven't run them yet) to
> see where we both stand. Let's examine the different parts:
>
> * JSON format: We're both using a format which is based on the scaladoc2 doc
> model and a tool which uses the scaladoc2 infrastructure up to and including
> the doc model. VScaladoc2 creates lots of individual files down to the
> method level.
>
> Extradoc has two modes here: One creates a single file for a documentation
> site. Structural redundancies are removed as much as possible (e.g.
> inherited methods are not represented by separate entities but instead their
> classes link directly to the original definition; VScaladoc2 seems to do the
> same) but little is done at the textual level (except for simple things like
> omitting definitions with empty array or string values which VScaladoc2 does
> not do). It is expected that such a scaladoc.json file will be gzipped (thus
> removing most of the remaining redundancy) and then used as an interchange
> format (e.g. for a GUI-based viewer, an IDE, or for merging multiple doc
> sites into one).
>
> The other mode (json-multi, also generated along with some static files in
> explorer more) is intended for tools like Extradoc Explorer which examine a
> single template at a time and can't be expected to download a complete
> scaladoc.json for that purpose. It currently generates one file per doc
> template (just like scaladoc) plus a separate index (global.json). In order
> to show a scaladoc-like presentation for a template, you will need at most
> the global index plus the pages for the template in question plus all
> templates in its linearization. Entities will be shared across the
> linearization but not outside of it, so there can be some duplicates. In the
> future, I'd like to merge smaller related templates (like sub-packages or
> class hierarchies) if they are too small for separate files. Having too many
> small files wastes space for local doc sites (fs block size) and on-line
> sites (HTTP request overhead; gzip compression). The json-multi format also
> has some further size optimizations like merged flags (e.g.
> "isPublic":true,"isDef":true,"isUseCase":true in json becomes "is":"udU" in
> json-multi). The extradoc.js script contains code to undo these during
> post-processing. All in all, json-multi is probably the part of Extradoc on
> which I spent most of the time.
>
> * Presentation: Is your web site already online? I had a look at the code
> and it seems this is where we're really doing different things. VScaladoc2
> has a Lift-based client-server app for generating a HTML presentation
> whereas Extradoc Explorer is completely browser-based. The more ambitious
> jdocs-like features of VScaladoc2 could not be achieved in such a way but it
> can be used locally just like a standard Scaladoc site. It makes extensive
> use of jQuery and bits of jQuery UI.
>
> The JSON files are loaded on demand via AJAX and all HTML is generated from
> them directly in the browser. The jQuery History plug-in is used to get a
> navigation system which resembles are normal web site or a local scaladoc
> site consisting of multiple HTML pages. All entities and views are
> represented in the hash tag of the URI, so you can bookmark them
> individually and use the browser's history.
>
> * Demo project: I don't have one, so I'll try to build yours with Extradoc.
> My standard test case is the Scala library because it's rather large and
> thus allows me to test performance and memory requirements.
>
> I think the most obvious part for collaboration at this point would be the
> JSON format and the tool that generates it. Maybe Extradoc could be used to
> generate the data for your web app. I think its JSON generator is quite
> mature by now and does lots of optimizations but the code needs to be
> cleaned up. It's still rather hackish. What do you think about Extradoc's
> single-file and multi-file JSON output? Would they be suitable for
> VScaladoc2 or is it better served by its one-file-per-method format?
>
>> PS: Your wed design is highly better than mine.
>
> Well, I took some design cues from Apple :)
>
> -sz
>