GENUKI implemented in the Drupal CMS

Content Management Systems

Drupal is a sophisticated Content Management System (CMS); this and related documents (County conversion pre-requisite tasks, Tasks required to convert a single county, and Checking conversion to current GENUKI implemantation) - which are intended for maintainers rather than ordinary users - are evolving documents that describe how Drupal is beginning to be used to implement a CMS-based website for what (just in this documentation) we refer to as current GENUKI implemantation. The document current GENUKI implemantation Maintainers Guide is an initial draft version of the documentation that will be made available for continuing use after the conversion is completed.

current GENUKI implemantation has a similar look and feel to the current GENUKI website, referred to here as our pre-Drupal implementation - we are gradually evolving our pre-Drupal implementation into current GENUKI implemantation, using a conversion process that is itself gradually evolving as we come across and find ways of dealing with variations and inconsistencies between the various parts of our pre-Drupal implementation. Since current GENUKI implemantation makes use of much more sophisticated tools for maintaining the web pages than we have been using for our pre-Drupal implementation, this is enabling us to create a framework that will make it possible to build a much more coherent website for, and to add much more functionality to, GENUKI in the future.

This document uses Drupal terminology as it provides a clearer description, and makes it much easier to relate to all the existing online Drupal documentation. The present version of this document, an overview of how we plan to use Drupal's features, and desription of the processes involved in converting our pre-Drupal implementation to current GENUKI implemantation, is aimed at existing GENUKI maintainers with no prior knowledge of Drupal - your assistance in improving and clarifying this document is sought.

The concept behind a CMS such as Drupal is that it is used to provide the management of a website's data (i.e. the website's "information content"), and the look and feel of the website, leaving the maintainer to concentrate on providing the information without needing to know the technical details of building a web page. As the information does appear in a web page, and you will have some concern for the fine details of how it appears, so you will want to define some of the mark-up. But you don't need to know the intricacies of HTML, and there are tools to provide other means of doing the mark-up. The other main difference is that the primary repository for your information is within the CMS, so you log on via the web to update it, rather than holding a local copy yourself. (However, we are investigating tools which would support maintainers who wish to hold their own master copies of their information content and upload this to Drupal.)

Note: Maintainers should in general use the Support Ticket facility to document and discuss maintenance problems, and proposals for new or changed maintenance facilities, rather than email. This will help to ensure that the issues that are raised are dealt with, that it is clear who is responsible for dealing with the issue, that we all can see what is going on and what needs to be done, and that we have a record of what has been done.

Themes

The overall look and feel and the basic framework of any web page displayed by Drupal is controlled by its "theme". There are many different themes already available in Drupal. So to start with we have created a GENUKI theme which is a sub-theme of one of the standard themes. More work is required on themeing, and with experience a more sophisticated basic theme may be provided. Indeed, there are much better ones which are much more flexible and which configure themselves according to the device being used, whether it is a phone, a tablet or a wide-screen desktop. The theme also defines the type of HTML being used, and we are going for HTML5 so that major changes should not be required in the future.

Work on the theme is taking place and a responsive theme is now in place, though there is still work to be done to investigate improving current GENUKI implemantation's usability from smart phones. Indeed, there is considerable scope for style changes and this could be investigated once the transition process has been successfully completed.

Nodes

The Drupal component that matches our existing pages is, in general, the "node". The node is constructed to hold the various pieces of information. Two initial node types have been defined, a Place node for all the standard county, town, parish, etc., pages, and a Plain node for all those further GENUKI pages created to hold additional information, where this information is rather too big to go into the parish page. (A third type of node, a GENUKI topic node, which relates to part of a page rather than a whole page, is described below.) 

As a (Drupal) maintainer you first login to Drupal, and you then have the ability to edit Place nodes and Plain nodes. When you have moved to the page that you wish to edit, you click on the edit button and so reach an "Edit screen" for this page. This shows you all the fields that comprise the particular type of node (Place or Plain), and provides "Help" information indicating how each field is used and is to be filled in. The initial fields in each node relate to such matters as page titles, contents listings, etc., which you need to scroll past to get to the field(s) where the actual data (information content) of the page is held.

For the parish, etc., Place nodes each of our standard topic texts ("Church Records", "History", etc.) is held in a separate field (a "topic box").  Each topic box can either be set to display HTML (when a blue command under the field is showing "Enable rich text"), or to show the contents of the field as it will be shown in the web page (when a blue command under the field is showing "Disable rich text"). This latter case is in fact providing you with a WYSIWYG - What You See Is What You Get - editing facility. (Here is some tutorial material on this editing facility.) You can flip between the two entry modes to suit your preference.  In WYSIWIG mode no knowledge of HTML is required. In the other mode (the HTML mode), only a limited set of safe HTML codes can be used. The allowable HTML codes are listed under the topic box.  (You can arrange that WYSIWYG is the default mode for you, for both Place and Plain Pages, by setting "Text Formats enabled for rich-text editing" to "Filtered HTML" and "GENUKI Topic" in the Edit pane of your Account page.)

In the Place node each topic box is identified by a topic heading. Thus the use of standard topic headings is enforced. Moreover Drupal ensures that the various topics will appear in the web page in the proper alphabetic order, and that their headings will be correctly listed at the top of the page in our standard two column contents listing format.

In the Place node there are also fields for the basic header such as the descriptive quote, and an optional image for the location map. There are also some fields (that the user doesn't see) which define where the web page that is created from the given Place node is in the GENUKI hierarchy, and for linking it with other information. However, it is the topic fields that are the ones you will use most.

The information that appears in every Place Page, such as map links, nearby places, and churches from the church database, is added to the place nodes automatically. This is achieved simply by providing a unique "place code" in each place node. For example the code for Lytham is LANLytham; this place code is recorded in column R of the gazetteer. The "place code" provides a simple method for Drupal to access information about the place with no complications of changing grid references or urls. (Phil has code that will automatically add a suitable value to the place code (column R) for a places.csv file, using the value in existing urls as a starting point.)

Each Plain node (such as the present node) similarly has various fields, with the actual information content of the web page being held just in the Body field. Other fields are again hidden from the user and are used to hold information such as the county, and the associated place (identified using the relevant place code).

In the future when we have a set of related information which could be best held as a set of fields, such as church pages from the church database, we can define new node types to help us manage them.

The nodes are set to make a new revision each time you edit them. If you make a mistake you can switch back to a previous revision, and you can compare revisions and see the changes that have been made.

The edit screens for the Place and Plain nodes are intended to be self-documenting, i.e. they contain explanatory information aimed at describing exactly how the various fields are to be used and filled in. Suggestions for improving these explanations are welcome.

Pathing and linking

As this is a CMS with the information content held in a database, we do not have a hierarchical file structure to provide urls to the web pages. The basic urls are in fact similar to those from other CMS-based sites and are of the form node/nnnnnn. However there is a facility whereby a friendly familiar (hierarchical) url can also be assigned to a node, and that is the "path". So the path defined for Lytham is big/eng/LAN/Lytham which is the same as its url has always been in our pre-Drupal implementation. Those for separate Topic Pages though will change, so that big/eng/LAN/Lytham/census.shtml will become big/eng/LAN/Lytham/census, as we lose the .HTML in the conversion to use Drupal. For non-HTML files such as images or pdfs, the name does not have to change, although there are facilities for uploading images directly when you don't know where they are, but you don't need to. As such files are much easier to manage using our existing techniques with a secure file transfer facility, such as WinScp (Windows) or Transport (Mac) it is proposed to continue to use them for non-HTML files. It has been found though that we need to map them to a slightly different path to avoid a problem, and so they are available as, for example, /files/big/eng/LAN/Lytham/StJohn.jpg - i.e. the existing url with "big" changed to "files".

There is also a facility (the "menu system") to define where a node sits in the user-perceived hierarchy. This automatically provides a breadcrumb trail at the top of the page so that users can see where they are in the hierarchy, and easily move upwards. It also gives us drop down menus, as a new means of navigation. However the basic theme that was initially chosen does not provide that ability, and is another reason to change it.

Images

Images  can be uploaded so as to available to appear within Plain pages, or (though this is less commonly done) within an appropriate topic field in a Place page. The uploading has to be done using a separate Secure File Transfer Protocol (SFTP) facility (e.g. WinScp for Windows, or Transport for Mac), rather than Drupal. The procedure is as described in our pre-Drupal implementation documentation on uploading. Images can be actually added to a page, either using the <img> tag when editing HTML, or by means of the Image icon (a computer screen with a small plus sign,with the hint 'Image' when you run the mouse over it) when using the WYSIWYG editor.

Users

Maintainers have a user name and password to logon to Drupal and gain access to the Maintainer facilities and documention, and are noted as the author of anything they create. As the author can be changed it is really the term for the (current) maintainer of that content. Some Drupal users are given special roles, such as 'GENUKI maintainers' - GENUKI maintainers can create nodes but only update the ones for which they are recorded as the author. We also have another role for some people, of 'GENUKI administrator', with more privileges such as the ability to edit nodes for which they are not noted as the author.

As the new Drupal-based system it is in effect a flat structure (for which we are providing a hierarchical facade) we could move away from current situation in GENUKI where a single person has to do everything for a given county. We could have people maintaining just a single parish, for instance, something that is now much simpler as all they have to do is work on the information content, not the layout. We could give them a role that just allows them to edit their own nodes, and not create any new ones.

Importing data to Drupal

Moving data from the current our pre-Drupal implementation system is relatively easy, since Phil has written a program that reads through a section (e.g. a county with all its parish pages, etc.) of our website looking at all the HTML files. For the parish pages it reads the subject headings to identify our standard topic headings and splits the page up at these points. It also identifies the page's initial descriptive quote and any image map at the top of the page. It removes the newlines from the sections, as they muck up csv files, and also converts any links within the site to the new naming convention. This is then saved as a csv file along with the relevant value of Place code from column R of the gazetteer. This csv file can then be imported into Drupal. There are options to only import new nodes, to replace nodes, or just work on specific pieces of content (update).

A visual check is then required to find any possible missing or garbled bits of information content - this will depend on how far from our standards the original our pre-Drupal implementation page layout is. (All the non-standard headings get flagged and Phil has to do work to amend the program to handle them, making adjustments when the program finds things it shouldn't, or if there is major style difference.)

The conversion and transfer is being done on a county by county basis, with a specialised team doing the specialised work and assisting the county maintainer.

If data is already held in a CMS format, such as Colin's counties, the import feature can be used to get the data initially into Drupal, and also to apply subsequent updates using the replace or update the information. We understand that Colin keeps individual topic items in separate files and uses a special program to assemble this infomation into HTML files. The code generation part of this program would require changing to place the data into separate fields in a csv file which can then be used by the import function. However these matters have yet to be investigated fully.

We can also use the import function to include substantial pieces of information that have come from other sources. Examples of this are the gazetteers which have been transcribed by Colin and Mel. These can be stored in csv files with identifying place codes, and then imported into Topic nodes. This is described in more detail further down this page in 'Views within a page'. This approach allows such information content to be managed quite separately and without any action from the maintainer of the county to which it relates, i.e. without the maintainer having to add the information into the fields of the edit screens of the county's various Place Pages. Instead, the relevant information just gets incorporated directly into the appropriate position in a Place Page when the user views this page.  This gives GENUKI a lot of additional flexibility and could even provide a mechanism where maintainers of other sites, such as Family History Societies, could provide linking information to them that gets automatically added into our Place Pages.

Searching

Probably the most useful feature that comes out of the box is the search facility (accessible using the search box in the top right hand corner of each page). We no longer have a local search engine (for current GENUKI implemantation), and Drupal has one built in that works well without any work for configuration or support. As there is a database at the back of everything there are facilities for searching and displaying the results using Views. This is very powerful and should enable us to provide additional ways of viewing and managing our data. (In practice you may find that your browser's Find command is also a convenient way of searching within the currently displayed page.) 

There is a large community working on Drupal and making code available. So if we have a new requirement it is quite likely that somebody else has already done the work, and we could implement that. For example, if we wanted to add online shopping, we would install the Ubercart module!

Views

Views are a powerful way of searching the database to provide either a complete web page, or part of the content of one. One has been constructed for current GENUKI implemantation, visible to maintainers, to search for and list nodes ("Select nodes and optionally perform bulk operations"). There is a link to this view from the Maintenance menu at the top of each page. You can specify criteria on which to search, and the order in which the results are displayed. For example, you can search for all the nodes for which you are the author, and have the results ordered by date updated. In the present facility it shows the node title, path and a teaser of the content to make it easier to see which of your many nodes they are.

Another of the search criteria is a character string, so you can find which nodes contain this string. Once you have selected some nodes, there is then an option to do a search and replace on this set of nodes.

Views within a page

One problem we have had in our pre-Drupal implementation arises when somebody transcribes a substantial work that covers a number of counties, such as a gazetteer, but then has difficulty persuading the county maintainers to add it in. Our current maintenance model is that the county maintainer is the only one who does the work for an individual county. So it would be useful if we could move to a model in which other people can supply and maintain some information that is included as part of the existing town/parish pages.

Drupal Views and Panes can provide this. A third node type, that of  of 'GENUKI Topic', has been defined where you can put in a single topic, for a single place. There are fields to record the Place code, the topic type, and optionally the topic source. In the Pane that displays the Place node, there is a view which searches for this additional information, and automatically displays it as part of the Place page alongside anything under that topic that may already be part of the Place node. (The facilities for creating this type of node have yet to be made generally available.)

So we can have people supplying and maintaining additional information that is automatically integrated as part of the Place Pages. Additionally we could also combine it in different ways. So if we did it for a gazetteer, we could see the appropriate section on the Place Page, but also have a page that showed all the places in a county, just as it would appear in the original gazetteer.

Gazetteer & Church database

The gazetteer is currently held as tables in a MySQL database, accessed via cgi scripts written in Perl. Any new access requires changes or additions to the scripts. It would be preferable to have them much more closely integrated with the web pages, and with an update model that did not rely on an overnight change to a csv file which is used to change the database.

This of course can be done with Drupal, but it would be best to delay work on this until all our pre-Drupal implementation's web pages have been moved to Drupal, as there would then be no requirement to have to have two systems live and updated simultaneously. A design choice would be required as to whether to keep to a distinct database table model, or whether to move to data stored in nodes. Nodes make it much easier to integrate with web pages, but takes us away from the csv file update model. But nodes make it much more flexible with regard to adding and maintaining data. It may be that we keep to the csv model for the gazetteer, but for churches a combined csv and node model may be more appropriate, the basic details being maintained via a csv file, with detail such as church history and records via nodes.

It does look as though we could add more features as well via the OpenLayers module which provides mapping support. This has not yet been looked at it in detail, but it appears we can plot things over other maps than just those from Google. We'd like to add the ability to plot churches on OpenStreetmap as that identifies individual buildings.

Access Control

This is a facility available to GENUKI Administrators, but not ordinary Maintainers, via the link "Access" near the foot of Edit templates. 

Other features we could add

As Drupal maintenance is via a userid model, we could add configuration features where maintainers could make changes directly, rather than having to ask for them to be done. At the moment we have a number of county options that are handled this way. It should also be possible to provide the ability to start Spider checks for a county when significant changes have been made.

It is envisioned that we would still run the spider to check links, and probably also to make HTML checks. A lot of the HTML is created by Drupal, but there are still a few bits that the maintainer can set so it is probably worthwhile retaining this check. But some bits of the spider can be removed.

Phil has put in place a blog about the current GENUKI implemantation development which would go if this goes live. However I think it would useful to have one for publicity and to describe any changes. A blog for counties may also be a useful feature.

Support Tickets

During the conversion process, when there is a need to report that a problem has been encountered, or that a potentially worthwhile new facility has been identified, rather than communicating this to one or more Administrators simply by direct email, please use the Support Ticket system. Items ("Tickets") entered into the Support system will be neatly catalogued, with an indication as to who is assigned to consider the reported issue, and of the state of the item: "new", "active", "pending" or "closed". (The displayed list of tickets can be set to show all tickets or just those of a particular type, e.g. "active".)  

At present four different classes of ticket have been identified, depending on whether the subject matter concerns "Drupal implementation", "Church database", "Gazetteer", or "Basic support". The priority of a ticket can be stated: "low", "normal", "high" and "critical".

To enter a new ticket into the Support System, use the link  "Post new support ticket" on the Support Ticket system page. This provides a simple form into the body of which details of the problem or suggestion can be entered, using WYSIWYG editing if required. At the bottom of the form are facilities for setting the state, priority and type of the ticket, and identifying to whom it is assigned. 

Tickets can be updated - the date of the last update is recorded automatically - and comments can be added.

Disciplined and systematic use of this ticket facility will we believe greatly help the administrators to cope efficiently with the considerable number of issues that can be expected to arise before the conversion is complete.

Miscellaneous minor issues

There is a problem with logging in using Safari on the Mac. When you move to another page after logging in, the page you reach sometimes shows you as logged out. The workaround is simple. When this happens, get your browser to refresh the page - you should then be shown as logged in.

Due to sloppy HTML the link text of an external link sometimes is not shown in blue and underlined (because it is missing the quotes round it), though the link is marked with a little box and up arrow, and does work. However if you open an Edit window on the page with this problem the WYSIWYG view of the page shows the link text complete with blue colouring and underlining. An effective and simple work-around for this problem is then to select this text and press the little link button (in the line of editing buttons), so that a window opens showing the Link Info. This information is correct, and by simply pressing OK, and then Save to return to the problem page, it will be found that the link text is now showing properly.

Though the links to newly converted pages, e.g. to parish pages from a county page should work correctly in dp.genuki as soon as the pages have been made live, the popup links in the blue banner will remain pointed at the original unconverted page until the pages are moved to www.genuki.org.uk.

Anchor Links

The conversion of pages containing anchor links has caused a number of difficulties and queries, as some weren't converted correctly. We do now have procedures in place that should prevent a re-occurrence of these problems (though there may still be some as yet unlocated and hence uncorrected issues with pages that were transferred before we understood and dealt with all the causes of the problems). This account provides details of the problems (and how they were solved), mainly just for the record.