Managing web sites with UserLand

Executive Summary

Frontier is UserLand's main product. It started as a commercial product for the Macintosh platform at the time when no good scripting language was available, was ported to Windows, was changed to a non-commercial product around the time when Apple came up with AppleScript, and went back to a commercial product following release 5. Release 7 came out in mid-2001. Release 5.0.1 can be found on several web sites, such as here.

Frontier is used to handle dynamic web sites contents, and consists in three parts: system-level verbs (set of APIs), a scripting language (UserTalk, similar to JavaScript and C), and a database (where web pages and scripts are actually stored.) Any application can interact with Frontier, including through the XML-RPC protocol. Contents can be added to the database through a dedicated client application (Radio), a form opened in a web browser (the Manila add-on), FTP, email, etc.

A common use of Frontier is to build blogs, a.k.a. web logs (online journals, diaries.) Due to its origin in previous Macintosh bestsellers ThinkTank and MORE also written by Dave Winer, contents are managed through an outliner, a tree where each element is a topic. A given topic can also have a note (also called a "paragraph") attached; This is often used to have the topic be the header, and the note be the actual contents of the topic.

Since Frontier was originally offered as a sophisticated scripting tool, it's no surprise that scripting is an important part of working with Frontier. Essentially, a Frontier web site consists in raw contents stored in the database and dynamically-enhanced by scripts before being delivered to web clients. Thanks to scripting, web pages can even be entirely rendered from scratch, and be either fed to a web server through a CGI call for higher performances, or handled by Frontier's integrated web server. This is made easier with Manila, a set of scripts that allow users to manage contents from a web browser.  Manila includes archiving, streaming news, permission-based updating and workflow tools, and is part of the Frontier package.

A third product is Radio, a light version of Frontier formerly known as Pike, without some of the Internet server features. It's used both to read news from remote sites through syndication with the RSS protocol, and also as a tool to create web sites. Radio is also built on the Frontier architecture, so all your scripts and web contents are saved in a local Frontier database.
Radio runs on the user's desktop, can act as a local web server, and can also be used to update a remote web server (through SOAP, XML-RPC, FTP, WebDAV.) Scripts can execute commands on other hosts through XML-RPC. Radio can also receive headlines from web sites on the Net that support RSS. Radio can also be used to edit XML files. Data can be edited in the Radio Windows client, or through the Internet Explorer browser (With either Opera 5.12 or Netscape 4.74, the HTML editing tools don't work.)

Drawbacks with using UserLand products:

Frontier

Note: Before rendering any web page (ie. having Frontier extract bits and compile them into an HTML file), make sure that the default browser is running, or Frontier will stall for about 30s before giving up.

A picture being worth a thousand words, here's a screenshot of Frontier 5.0.1 for Windows, which displays the root database (frontier.root), where everything lives, and a sample of a web page (websites.myFirstSite.#prefs.default):

The main table in the Frontier database is called the "root"; It lives in Frontier.root, and is always loaded while Frontier is running. If the content of the database is not displayed, select Windows | Frontier.root to display it.

To create a backup of the Frontier.root database, select Main | Backup... . A new backup is created in Frontier\backups. If other backups are already located in that directory, Frontier increments a counter to create a new filename, eg. Frontier.0.root, Frontier.1.root, so it's safe to only keep the latest and greatest. If Frontier.root is corrupted, stop Frontier, rename or delete Frontier.root, rename one of the backups to Frontier.root an move it to the directory where Frontier.exe lives, and start Frontier.

A Frontier database can contain different kinds of objects, including other tables, formated text ("wp text", ie. word-processing text), scripts, strings, numbers, etc. An item in the database can be opened by double-clicking on it, by putting the focus on the item and hitting CTRL + the Enter key on the keypard, or CTRL-F2. Any object that you create is added to websites.#data.

Items in the root database are organized and refered to hierarchically. For instance, the "backups" sub-section under the "user" section is refered to as user.backups. You can use Main | Jump to jump directly to a section or item. A sub-table can have its content opened in an independent window by selecting it in the root database, and double-clicking on its Value column. To select a whole line, click on either its Value or Kind column. To edit the name of the time, either click on it in the Name colum, or hit the Enter key on the keypad.

To "publish" a web page, ie. extract it from the Frontier database and save it as an HTML file, select the item in the database (eg. websites.samples.davenet.aboutThisWebsite), and choose Web | Publish Page: The default web browser is launched to display the output, and the HTML page is located in Frontier\Websites .

Before being rendered, all web pages prepend the text that is included in websites.#template . Post-rendering customizations are achieved through filters. You can build a list of Web sites whose URL you often use by adding them in user.html.glossary . Any item that starts with # is called a directive. You can either use the top-most directives that live in user.prefs and are common to all sites, site-specific directives (eg. websites.myFirstSite.#prefs), or page-specific directives in a web page (eg. add #autoparagraphs "true" at the beginning of websites.myFirstSite.#prefs.default).

The #autoParagraphs directive adds <P> tags automatically when it finds two consecutive returns. The #clayCompatibility directive surrounds any line that starts with *** with <B> and </B>, and remove the three stars. Default settings can be changed in user.html.prefs . A directive must start at the first character in a text, ie. there must be no space or TAB character before the directive.

In an outline, use CTRL-D, CTRL-U, CTRL-L, and CTRL-R to move items down, up, left, and right, respectively.

The #template section represents the framework through which all web pages will be rendered; It's the right place to add headers and footers that have to be added automatically in all pages on your web site. By default, #template is an outline, but it could also be a wp-text. The template must include a {bodytext} section where the document to be rendered will be inserted. You can add {pageheader (title)} as the first item in this table to get Frontier to look up a "title" item in your site and set it as the page's title (ie. add a <TITLE></TITLE> section to the rendered page.) As a final touch, you could add a {pagefoot ()} as the last item in the table to have Frontier add eg. a copyright section. Site-wide templates can be added in user.html.templates and be referred to in web pages with eg. #template "normal" .

When using a basic outline to create a Web page, idented items are automatically rendered as unnumbered lists. When adding the #renderOutlineWith "prettyOutline" directive, however, top-level items are rendered into upper-case H3 headers, lower-level items are turned into H4 and numbered, and a horizontal line is added between each top-level section. Building an HTML-filled outline with #renderOutlineWith "newCulture" renders the document as pure HTML. Renderers are actually UserTalk scripts, which can live in either user.html.renderers or your site's #tools section.

Macros can be included in a document by using curly braces, eg. {3+4}, {user.prefs.name}, {clock.now()}, {renderObject(@websites.myFirstSite.default)}, etc. If the macro refers to objects that live in the database, it looks successively in the #tools section of the site where the rendered page lives, followed by user.html.macros, system.verbs.builtins.html.data.standardMacros, and finally in the table pointed to by ??.html.data.adrPageTable. If all else fails, it looks in the entire database. As an example, the pageHeader() macro lives in system.verbs.builtins.html.data.standardMacros.pageheader.

Here's an example of the use of macros in an outline object:

To add an image file into the database, select any item in your site table, and add a sub-table using Table | New Sub-table. Name it "images" (the "s" is important). Next, select this new table (without opening it!), select Web | Load Image File, select any image file on your hard disk, and name it eg. "myimage": The image file is added to the websites.myFirstSite.images table. It can now be refered to in any page using eg. {imageRef ("myimage", explanation:"Logo", align:"middle")}. Note that if you keep all images in the default user.html.images table, you can refer to them using eg. {imageRef (@user.html.images.myimage)}, and when an HTML page is rendered, pictures will be saved in the same directory as the HTML pages instead of being saved in ../images/ . Likewise, you can add a background picture by saving it into the images sub-table of your site, and refering to as #background "myimage".

Shortcuts ("Glossary") such as HTML hyperlinks can be included by adding entries in user.html.glossary (eg. Frontier = <a href="http://www.scripting.com/frontier/">Frontier</a>), and adding a reference in an HTML document by enclosing this shortcut between quotes (eg. This site was built using "Frontier"): When rendering the document, Frontier will look up the entry in the Glossary, and replace the string with the hyperlink. Another way to achieve the same thing is to use a macro: I have {glossSub("Frontier", "a really cool tool")} for you. If you'd rather have your own Glossary table instead of using the default user.html.glossary, use the #glossary directive to tell Frontier where to look up shortcuts.

If you'd rather do away with the Glossary, you can define shortcuts right in your document (outline only):   #define "UserLAdd" UserLand Software, Inc.<BR>P.O. Box 1218<BR>Burlingame, CA 94011-1218<P>, where the second part is a child item of the parent #define. Next, add a reference anywhere in your outline using {MyUserLAdd} . If your document is not an outline, use #myTitle "This Is A Cool Title" and reference it with {myTitle}.

Since Frontier is just replacing one string with the other, the Glossary can be used for shorthanding any repetitive sentence, eg. UserLAdd could be replace with "UserLand Software, Inc.<BR>P.O. Box 1218<BR>Burlingame, CA 94011-1218<P>". To disable use of the Glossary, set the #expandGlossaryItems to "false".

You can take advantage of filters to tell Frontier to perform additional tasks before rendering documents. Filters live in your site's #filters sub-table, where you'll find default objects like FirstFilter or PageFilter.

A site can be rendered ("published") in one go. More recent of Frontier are supposed to be able to render and upload a site directly to a remote FTP server, but the free 5.0.1 version cannot: Frontier only publishes a site on your hard disk, and it's up to you to upload the files to an FTP server.

The default reference to the server where the site will be saved is set in websites.#ftpSite, but you can use a site-specific reference under the same name, eg. websites.myFirstSite.#ftpsite. The Folder sub-object is the name of the directory where files will be saved locally by Frontier after they are rendered, isLocal can be left to false since Frontier can't call an external FTP client, and the url object must be set to a valid URL, eg. http://intranet/. Other objects that are needed by FTP-capable versions of Frontier include account, directory, domain, and password.

If you want to create sub-folders so that not all HTML rendered files live in the root level, create a sub-table with Table | Sub-Table, and create a new outline or wp-text object inside this new table. If a document includes a reference to another document but you wish both documents to be rendered as one file instead of two independent files, just move the referenced file into the #tools section, and update the reference to point to this new location: Files in the #tools, #glossary, or images folders can be referenced but will not be rendered as independent files.

A cool way to include a list of all other files in your site is to add "{outlineSite ()}" in eg. the main document of your site.

file://localhost/D:/Download/Frontier 5.0.1/Website Tutorial/SiteOutlineandNextPrev.html

To create a new web site, select Web | New Website..., and give it a name.  It will be added to the websites section in the database.

Frontier User's Guide http://www.scripting.com/frontier/manual/default.html

Manila

Manila uses a feature called cookies to identify members. Since some browsers let you turn cookies off, make sure they are on before you begin.

If you like, you can create a new Home Page for daily updates instead of editing the same one. This is called "flipping" the Home Page. When a Home Page is flipped, the old one is archived and the date it was created becomes a link on the Calendar. Clicking on a date calls up the archived Home Page. The month links at the bottom let you move between the months in the archive. The Calendar only appears on the Home Page and Discussion Group pages, both of which can have archives.

The cowskull is a link that calls up your Home Page as a DG (Discussion Group) Message.

Radio

Playing with Radio 8

Although I'd rather that Radio be used like CityDesks only as a sophisticated tool to update a local or remote web site, I'm not ready to give up on it. Radio is a strange beast: It is both a dedicated Windows/Mac client + web server that lets the user work on his local site (.root files?) through a web browser instead of through the Radio client + receives news items through XML-RPC + creates news items in the local site with possible replication on a remote site + macro language UserTalk.

After installing Radio, the install program launches Internet Explorer to connect to Radio's local web server, and prompts you to create an account on UserLand's server to upload your weblog for free (eg. http://radio.weblogs.com/0100000/). Apparently, there is no way to not create an account. As explained in the Prefs section, Upstreaming sub-section, Radio updates the copy of you weblog that sites on UserLand's server every 10 seconds.

Radio's web interface

Radio's native client interface

This is where the real fun begins, because Radio is not just a cool weblogging tool, but also a full-fledged development tool to build Web-based applications.

To launch the native client, right-click on Radio's icon that is now running in the taskbar , and select Open Radio.

Development

When creating web pages from scratch in the www/ directory, you can define directives (ie. variables) in a filethrough #mydirective "Hello world", and reference them later using <%mydirective%> . Longer directives can be saved as files (eg. #itemTemplate.txt). Directives can also be built-in verbs from the Frontier programming environment, eg. "The current date and time: <%clock.now ()%>." As you can see, you can embed code in web pages by using <% and %> to tell Radio where your code starts and ends, respectively.

Old stuff

Launch Radio, and connect to it through http://127.0.0.1:5335:

Older testing

As of October 2001, Radio UserLand is a free (as in free beer) application that is meant to run on a user's desktop host. It comes with a Frontier database and a web server, through which you can read syndicated news from RSS-capable web sites (the Home page below), and manage your weblog (Weblog item below) that users can either read by connecting to your computer where Radio is running or by uploading whatever files and directories are rendered by Radio into Drive:\Program Files\Radio UserLand\www\weblog\ .

In other words, Radio is a free, although proprietary, equivalent to GreyMatter or MoveableType where data would be saved in a single-file database and be rendered into a web site by running a scripting language that comes with the program.

In Radio, data and scripts are saved in a local Frontier database. Its content can be accessed either through the Radio dedicated client, or by connecting to Radio's web server through a web browser.

Here's the list of sections that you find in the toolbar in the Radio web pages:

Reading news

To view the current headlines, launch Radio, and connect to the home page of its web server through http://127.0.0.1:8080/UserLand. By default, Radio downloads new headlines automatically, but you can force a refresh through the Radio dedicated client by selecting Tools | MyUserLand | Do Scan Now.

Only those servers to which you subscribe show up in the home page. To subscribe and unsubscribe to RSS-capable web sites, click on the Subscriptions button at the top of the page. Clicking on the POST link at the right of any headline lets you add it as a new item to your weblog; You can add your own comments to let user known why this headline is important, and users can clik on the original URL to view the rest of the article on the original web server. Data are saved in the myUserLandData.root database, in the Stories section.

Creating sites

The Weblog item at the top of Radio web pages is used to create either brand new stories, or add comments to headlines that are displayed in the home page. As of Oct 2001, editing of your web log can only be done in IE 5 (don't know if it's because it's apparently the only one that supports DHTML edit boxes.) The HTML buttons don't work in either Opera or Netscape.

To tell Radio where to upload the rendered web pages of your weblog, scroll down to the Mirroring via FTP section, and set the required parameters. Unfortunately, the password is erase every time. Other ways to upload contents is through the Mirroring to a Manila site or Mail-to-Weblog sections. Needless to say, the former requires that you have an account with a remote Manila-capable web server. Yet another way to upload contents is through upstreaming (Prefs | Upstreaming server configuration); By default, contents are uploaded to http://www.ourfavoritesongs.com, but it can be changed.

You can also create documents directly in the Radio client through File | New. Such files are saved as independent files (ie. not in any of the *.root databases that come with Radio) in the OPML format. Another way is to create a WP-Text item to the websites.#prefs section of Radio.root followed by Table | New Text, but I haven't how to render it into HTML.

Issues

Why three Frontier databases: Radio.root, MyUserLand.root, myUserLandData.root?

On to create a whole site directly in Radio and publish it to a remote web server?

Appart from creating and saving a single file through File | Save as HTML, I haven't found a way to create a site in Radio, and render the entire thing into a directory tree located on a remote web server. This is available in Frontier 5.x above, and I assume more recent versions of it, but there is no such thing as Web | Publish Table in Radio.

No documentation ships with Radio. Sparce documentation available from http://radio.userland.com/directory/3541, but no tutorial for newbies.

Other sources of info are http://radiodiscuss.userland.com/directory/3541 and http://radiodiscuss.userland.com/discuss/ .

Radio is a bit puzzling for newbies, because it's both a tool to let you publish your own articles and also receive news from remote sites to which you subscribed. The home page is the latest headlines that Radio grabbed through RSS. If you want to add a reference to those headlines and add a comment in your own weblog, click on the POST button on the right of the headline.

Another puzzling thing is that documentation tells you how to add/edit weblog entries by connecting to the Radio web server running on your computer, but doesn't say where those data are saved in the Radio.root database, and how to view the result. Turns out that once you have hit the Post & Publish button in the browser, the rendered HTML files live in the www/ sub-directory, but those rendered pages are not accessible through Radio's web server (have to drag-n-drop them in your browser.)

Publishing your weblog means that your changes will appear on UserLand sites like Our Favorite Songs, UserLand, etc.

After launching Radio, a web server can be reached through http://localhost:8080/UserLand/ (possibly port 80 if no web server is already running.) Rendered weblog pages live in C:\Program Files\Radio UserLand\www\ . To change your weblog's appearance, go to http://127.0.0.1:8080/UserLand/weblog.wsf, scroll down to Themes..., select a different theme, click on Submit, scroll back to the beginning of the page, and click on Post & Publish. To check the result, go to http://127.0.0.1:8080/weblog/ .

Several databases can be accessed through the Windows menu: Radio.root, MyUserLand.root, and MyUserLandData.root. To refresh the contents of the headlines of the web sites to which you subscribed: Tools | MyUserLand | Do Scan Now, and hit Tools | MyUserLand | View in browser.

You can subscribe to web sites on the Internet, and have headlines be shown in Radio thanks to XML-based format RSS (Rich Site Summaries.) The headlines from the different you subscribed to show up as the welcome page, eg. http://127.0.0.1:8080/UserLand. You can delete any headline by checking it on the left, and hitting the Delete button at the top of the page. To add a headline to your weblog, click on the POST link on the right of the headline, and add your comment.

To create a new web page, select File | New, give a title to this page, which will be used as the page title, both in the browser's title bar and the page's title, so you should use spaces and upper-case; The file will be saved with the extension .OPLM . Add items in the outline, using HTML tags in the HTML menu. You can hide most HTML tags by selecting HTML | Format Text; Heading tags are displayed at all times. To display the HTML output, select File | View in browser.

Writing scripts in UserTalk

FAQ

RSS

What to do about RSS?

XML-RPC & SOAP

From the Dave Winer interview: XML-RPC (Remote Procedure Call via XML) and SOAP (Simple Object Access Protocol)  are used to implement publish-and-subscribe in Radio, so that you can receive notification when any of the RSS feeds you're subscribed to have changed. (Note: only if they support the <cloud> element in RSS 0.92.)

OPML

From the Dave Winer interview: OPML is the native file format for Radio's outliner, which is used to author Web pages on Manila sites, and directories and slideshows, and is accessed through SOAP and XML-RPC. Manila is the most completely Internet-scriptable application in existence today, and Radio is wired right into it, through SOAP and XML-RPC, of course.

Post. Publish. Who needs both?

Posting means that an item that you created through Radio's web server is saved in the Frontier database. Publishing means that this item is rendered into an external, HTML document, that you can then upload to a web server.

Resources