GENUKI Maintainers' Pages

Version 2.2

Broken links, Re-directs and other problems


The spider is run at the start of each calendar month and its output is stored on a database held at Maintainers can request that the system administrator run the spider at other than the monthly intervals. For reporting and maintenance purposes, GENUKI pages are contained in logical sections and the spider can be run against a subset, including one, of the GENUKI sections.


A short explanation of how the spider works is available elsewhere.


Spider reports required by maintainers are generated when requested by means of cgi scripts which read the database and create a web page containing the required data. The data in each report is therefore the most recent available at the time the report is requested.

The most recent spider report is available as a normal web page, but if a more up to date report is needed, it can be refreshed by invoking the spider cgi script.

Several different spider reports each containing a different format, and content, are available as follows:

Because the various reports are contained within web pages generated dynamically from a database they can take a noticable amount of time to complete depending on the server load at the time. Most reports include links to subordinate reports which are also dynamically generated. Returning to a superior report from a subordinate report web page by using the "back button" on a browser will therefore cause the superior report to be regenerated dynamically a second time. Users of the dynamically-generated overall spider report are therefore recommended to use the browser option to open web pages in a new window or new tab to avoid this regeneration delay. For the same reason, it might be worth saving a "web archive" copy of the spider report page(s). Alternatively, use the static web page which is a copy of the most recent dynamically-generated overall spider report.

Overall Report

The overall report is generated by invoking the spider cgi script. The resulting web page is organised by the logical sections in which GENUKI pages are organised for maintenance purposes:

Each line in the report consists of the following fields:

Section Report

The section report provides a summary of the section as follows:

Section Files Report

The section files report provides a list of the urls of the requested files. The file types requested can be one of several alternatives:

Redirects of genuki pages are also reported here for:

The section report also includes the section problems report for all problems in the section.

Section Problems Report

The section problems report provides a list of the urls of the problem files. The problem types requested can be one of several alternatives (see the headings on the columns of the Overall Report):

The resulting web page can be sorted into several different orders using the sort order directives at the top of the page:

Correcting Errors

Maintainers are expected to deal promptly with the errors highlighted in the reports, and it is recommended that maintainers take a structured approach to dealing with and correcting the problems notified by the spider.

1 - Confirm the error

Because a timeout error, or even a redirect error, detected by the spider's analysis might only be temporary, some errors documented in the spider reports might have disappeared by the time the maintainer looks at the reports. Before spending a significant amount of time dealing with the causes of certain types of errors, particularly but not just timeouts, maintainers should verify the errors are still valid.

2 - "404 Not Found" errors

The spider starts from a position of knowledge of all the pages that constitute the complete GENUKI website and these get checked individually on each spider run. New pages added to GENUKI since the last spider run are detected during checking and, at the end of a current run, these are added to the list of pages that constitute the complete GENUKI website. These pages are checked on the next spider run.

But when web pages which are on the list of pages that constitute the complete GENUKI website have disappeared, either because they were deleted deliberately or because of a problem, the spider cannot find them. In either situation these pages appear as a "404 Not Found" error in the "Page" section of spider problems.

If the spider simply deleted the missing web page from the list of pages that constitute the complete GENUKI website, then maintainers wouldn't be alerted to pages that had gone missing because of a real problem. Therefore, these missing pages are indicated by a button marked by a red icon with an 'X' on it. Clicking this button deletes all references to the problem as well as the reference to the missing page. This action must be taken for each GENUKI web page that has been deleted deliberately. If the web page was deleted by mistake then it will need to be reloaded.

There are no authorisation checks against use of this button, and any maintainer can use any such 'delete' button. So, please don't use it on pages for which you aren't responsible.

3 - Link errors and re-directs

Maintainers should check for, and correct:

For each re-direct error:

4 - Check before uploading

It's often hard to remember to check for errors and make the necessary changes to correct them. And if your county is fairly quiet, the time between new additions and changes might become extended. The dates and colours in the spider output can be useful in reminding maintainers of just how long it's been since they conducted their regular GENUKI maintenance tasks.

Before uploading your pages to the web it's wise to check all links. If you choose to run a link checker, either on your own computer, or using a service available on the internet, there is further advice available on which products and services are available on the maintenance software page.

If you've checked all your links before uploading web pages then it's unlikely that the spider will detect any errors.

Redirects are flagged as a problem because they often indicate a broken link that has been hidden. When a web site has been restructured, it is common practice, often on Local Government web sites, to use redirects to point incoming browsers to replacement pages in order to avoid reporting a broken link to the user.

In these cases, the redirects will be removed at some future date and this will result in real broken links. So, it is wise to examine redirects and check if they represent pages that have been moved to a new location. Correcting the link by using the new location will avoid having to correct a broken link in future when the redirect is removed.

However, not all redirects fall into this category, and some redirects are genuine and have been provided for a good reason.

[Last updated at 15.09 on Monday, 18 August 2014, by Mel Lockie. 2014]
Hosted by Mythic Beasts Ltd.