Hide

Genuki conversion from Drupal 7 to Drupal 8

hide
Hide

Help and Guidance 2020:    Current:  Modified Page: Version 1

HISTORICAL DOCUMENT 

Hide

Introduction

Genuki is built upon the Drupal 7 CMS and whilst Drupal 8 was nearing release when we converting to Drupal, Drupal 8 had not reached sufficient stability and it was lacking in features so it was not appropriate at the point. Since then Drupal 8 has reached maturity and we need to move to it before support for Drupal 7 ceases which will be in 2021. Drupal 9 appears to be scheduled for release around that time and as it will not be such a significant change as that from 7 to 8  move from 7 to 9 has been suggested within the drupal community. So whilst this plan is for the transition from 7 to 8 as we near migration time we could well be considering moving directly to drupal 9.

This describes our plan for moving our service to Drupal 8. As the internal technology has significantly changed in Drupal 8 the change requires a migration of our data to the new system rather than just a software change. It will also require us to re-write our own code which we have added to our Drupal system.

During our initial conversion to Drupal and running it in service we have learnt quite a bit about Drupal and have realised our implementation has some areas that could have been done in a different way, so as we need to undertake a migration we should consider whether we can make changes as part of the process so that we internally do things in a better way and to provide a simpler interface for maintainers.

Pre-requisites

Drupal is written in PHP and the version we currently run for Drupal 7 is PHP 5.4 whilst Drupal 8 requires PHP 7. So we need to upgrade the core of our Drupal 7 system so that it works correctly with PHP 7. We then need to modify the operating system so that it contains PHP 7 rather than PHP 5.4. Both these changes will require service downtime. Both completed.

Running two versions of Drupal simultaneously

In order to run both systems together on the same server we will need to have second Drupal database with a different name and access our test system via a different url, probably dp.genuki.uk again. The test system would not be accessible via https:. When we are ready for conversion it will be a single step process and after completion the urls will be swapped. We cannot convert on a county by county basis. Once we are ready to migrate to drupal 8 we will produce a third instance of drupal so that we can migrate into a clean system without any clutter that may left over from the development. That will give us quite a clean system. Once we have moved to drupal 8 we should also keep in place a development system. That will enable us to test fixes and develop new code without impacting our live system.

Access to Drupal 8 development system

There will be no access restrictions for ordinary users, but we won't tell anybody about it as it won't contain any user data. We also won't have maintainer logins until shortly before it is ready to release. We will though provide logins for the Genuki admin team so they can see how we are progressing, and also to be able to see and comment on any changes being made to the way we do things.

Migration

A migration is required to move data from 7 to 8 and this will result in changes to node ids and taxonomy ids. As we use these extensively in our data we will .likely need to add some temporary fields to these structures to temporarily hold the values. That will enable us to use them to search for the correct entities when adding in the new nids and tids.

One thing that is likely to be lost is node revisions as that is an added layer of complexity that we could do without. We do need to ensure there is sufficient time between announcing the migration date and it taking place so that significant changes to nodes aren't made just prior to it.

Fields

Fields to remove

Navbar - This was to hold the up arrow links we had at the top of every page which in most cases have been replaced by the breadcrumb trail. Just a few had more than one up arrow and Colin still seems to have up arrows on most of his nodes although the bread crumb provides that functionality as well. We should remove this field and for those few cases needing something here, the up arrows could be moved to the Introduction field.

Location map + Location image - These were used before we moved to using the media module to handle images and they are no longer used although the code does seem to be able to handle some changes. E.g. for WRY Colin seems to not specify an image and expects one to magically appear. (There is nothing in the field importer to do it for him either). So I can temporarily fix it for him by changing this field until it goes on his next update.

Exclude title from display - Only on plain nodes. Added to make Devon conversion easier and was a quick fix for a problem. Now that we don't have the time pressure of conversion it would be useful to consider adjusting the titles of the nodes using this and remove this feature to simplify things for ordinary maintainers.

Fields to replace

Place code - The place code was introduced as a replacement for instances where a url or a grid reference was used in the past to link items of data together. Changes to either of these broke the link and so the place code was introduced as a fixed key to overcome this problem. It is however a little clumsy as when you want to make a change you need to manually search for the new code and spell it correctly when changing that field. We do have a better way of doing things now our data is in drupal, the entity reference, which we now use to link place nodes to gazetteer entries. It makes things more efficient as it doesn't require a search whenever that link is used and we can provide a search interface for changing the link. We have in effect already replaced it in place nodes, but it is still currently used in plain, church and topic nodes where it used used to link to the place node. Now we also have a 'Parent place' field in place nodes that was added to handle places beneath parishes. This field should be added as a replacement for the place code in plain, church and topic nodes and we would then have a consistent field across our node types. Now to avoid a lot of work and complexity we could add code when a node is saved so that if it is empty it would use the url to work out the parent and add in the entity reference to that. It would still be possible to add something different if that was required. This could all be achieved as part of the migration to drupal 8. We would still retain the place code for now in gazetteer entries if there was still a need for it, but there probably is no longer a need for it.

Topic fields

One thing we should consider is whether to change the way we hold topic data, to make the job more flexible and to help simplify maintenance whilst not changing what the user sees. It is little technical but we could make editing a little simpler and speed up some of the things that take a long time. In Genuki-1 that only unit of data that we held was the web page labelled by the url and within which we had text which should have followed our standards with a fixed set of topic headings. Visually we could separate the different topic items, but it wasn't easy to get the computer to do that due to differences in which each maintainer did things.

So we moved on a bit when we converted to drupal as the equivalent of our old web page was a box called a node and within that box were more boxes called fields. these boxes contain the pieces of data.  So in each place node we have nearly eighty different boxes for each of the topic types that we support. Each topic has its own special box that contains the data for that topic, such as the census field, the 'archives & libraries' field etc . This does mean that most of the boxes are empty but we do need to look in each one when processing a node to see if there is anything in it. This model does not let us just have the boxes we need for that particular node and they are all visible when editing a place node which is rather a mess. Each of these boxes is of a specific type for that topic and any code that we write has to have eighty different variations to handle each type.

Now plain nodes just have a single box to hold data and that box type is 'body', it is not a specific type of box for the type of topic that may be in the plain node. So help label up that data we have another box next to it that uses or Topics taxonomy which we can set to specify which sort of information is within the body. Of course that needs to be set manually as there was no way to automate that when we moved from genuki-1. It is also a little restrictive in that whilst we may use a plain node to hold a single document, we can only associate it with a single topic type even if it may contain items of information related to more than one topic type.

There was no time to devise a better model as we were in the midst of the long conversion task but as the church database conversion was taking place later there was time to find a better way of holding topic data in church nodes. This was achieved by using an add on module providing field collections with which we were able to define a topic box which initially just contained a pair of boxes, one to contain the data and another to contain a topic taxonomy entry as had been done for place nodes. Now drupal lets you have multiple instances of a specific field so we could start with just a single topic within a church node and keep adding more as more topics were required within that node. This eliminated the need to have the eighty or so boxes required to let us use any of our topic types thus making things a lot smaller, more efficient and only the topics being used are seen in the edit form.

This worked well and so this field collection was used in topic nodes and thus enabled us to have more than one topic type within the node. When using topic nodes for gazetteer entries it was realised that further labelling options would be useful so that the data could be kept separate from the labelling, in this case to hold the name of the gazetteer, so an additional box to hold a sub-topic was added, and a plain text box so that maintainers could also have a box to do there own sub-labelling. A further possibility, would be a field to hold a date relating to the data although this has not been implemented. This opens up future possibilities for selecting and sorting the data items we hold.

Now if this model was used for place and plain nodes it would remove the need for eighty or so field types, make the edit screen considerably smaller, make our locally written code simpler and should speed things up. We have an opportunity to do this in the move to drupal 8 as the change in structure could be achieved as part of the migration process. There are always some issues which might detract from a scheme, one of them being that not all other modules support field collections. We had to add code to search & replace to make that work. Since we started using field collections similar functionality has been provided by different additional code called paragraphs and this does seem to be being developed well. So as part of the work to see whether to change the way we handle topics internally will be to look at paragraphs as well. Then once we have had some experience of  the fine details of how things may work, we will be in a better position to make some design choices.

Theming

One of the first we will need to work on are the themes which define the regions within the pages presented to the user and the look and feel as this is where some of the css classes are defined. SO we need this in place before working on the code as we will need to be able to work on the look and feel as we progress. The page regions don't really affect the maintainer experience as our data is presented in the Content region. We will need to work on a theme used for the data the user sees and an administration theme which is used when a maintainer is editing nodes etc. Drupal uses a separate light admin theme so that it is possible to fix things if a problem arises in the normal theme. Currently our main theme, called Genuki2 as it is our second Genuki drupal theme is a sub-theme of Zen. That was chosen as it seemed to be the most popular one at the time. For drupal 8  it will be based on Bootstrap as that is one that is now frequently recommended, and is designed to be responsive handling both large and small screens. We currently use Adminimal as our base admin theme with very little changes in our sub-theme.

It was recently suggested we change the top blue bar in our main theme so that it is fixed at the top of the window with only the bit below scrollable. We will need to think about that when designing our main theme as it does have a disadvantage in removing part of the screen estate which could be a problem when viewing on a small screen such as a phone.  With regards the blue bar it may be useful to make some change here. There is also an admin bar which currently all Genuki administrators now see. The maintenance menus will be moved from the normal blue bar into the admin bar, and make it visible to maintainers.  The admin bar also contains shortcuts and we can thus move all the administration into a fixed bar at the top of the screen.

Within the theming quite a number of css classes are associated with the various items that appear within our pages, and a lot of them are there as part of a basic template in case we need to use them. This produces quite a lot of clutter in the html used for our pages. We should try and see if this can be reduced which will reduce the amount of html within our pages making them faster to load and easier for us to find things when we do need to look at the raw html.

Disk space

We are nearing our limit on disk space for backup and we may need to do some tidying up before we start on the drupal 8 development. We cannot exclude that from the backup as most will be within the mysql database. We will exceed our backup limit during migration from drupal 7 to drupal 8 which should be for just a few weeks so we may be able to negotiate something with Mythic Beasts as we approach that time.

Configuration

  • The drupal 8 system can be accessed at http://test.genuki.uk/
  • This instance of drupal is stored on disk at /drupal/8-dev  Change to that directory when using composer commands to change the system. TYpe drush9 to invoke drush as plain drush will invokes drush version 8 which we use for Drupal 7.
  • css changes should be made in themes/custom/genuki/css/genuki.css

Search

We use Apache Solr to provide a search engine. Solr is a java application that is not itself a part of Drupal. There are a number of ways it could be configured, and many suggestions on the web, some with a working recipe but most are incomplete and some suggest ways to use the commands but you have to guess which ones contain parameters required to get it work. Our configuration details have been documented so that if changes are needed nothing gets forgotten.