Renovating A Web-site

Renovating a Website

Plus ça change, plus c'est la même chose

This website had become so dilapidated that it was well overdue for renovation. There were scripts that didn't actually work, and many of the processes that gathered browser user-agent strings were obsolete or in need of maintenance. After I commenced the project it evolved into something more than a "website renovation" diary. It became a case study in maintaining a twenty-year-old website in the age of AI.

### Cleaning Up Link Rot

The PGTS site was afflicted with a serious case of Link Rot, affecting both internal and external URLs. I decided to address this first. Most of the content on the site has been created manually with "vi". I was not inclined to modify each broken link by hand. Instead, I decided to automate the repair. I decided that the repair process would be as follows:

Consolidate and convert existing internal links: I created a perl script that searched for links to http://www.pgts.com.au and converted them to https://pgts.com.au, rather than re-direct the, using apache. I managed to get most of them in the first phase, but had to dig a bit deeper into "included" PHP and perl code to find, fix and/or remove all of the legacy cruft.
Search for broken links: I decided to use "checkbot" to find all broken links. The TEST version of the site runs on an ancient HP Proliant server tower, which, many years ago, was the PROD server. Since there is only one webhost and one desktop in the TEST region, I don't use a TEST DNS. Instead, I override lookup of the PGTS webhost with /etc/hosts on the TEST desktop.

Checkbot seems to have been included as an add-in for Chrome and there are lots of clones. I opted for the original, as a command-line version of the bot would give better control than a browser add-in. There is a version on SourceForge and an RPM on OpenSuse. I downloaded the tarball from SourceForge. I discovered that checkbot may require a little maintenance. When I tried to "make" the target I got the following:
```
 Bareword "LWP::Version" not allowed while "strict subs" in use at ./checkbot line 530.
 Execution of ./checkbot aborted due to compilation errors.
```
The README file in the downloaded package mentioned Hans de Graaff as the author; however, the URL quoted in the documentation was broken. So I assumed that his email address would also be obsolete. I examined the code. The fix turned out to be surprisingly simple ... Line 530 read as follows:
```
 $main::ua->agent("Checkbot/$VERSION LWP/" . LWP::Version);
```
So, I "patched" the code by replacing LWP::Version with $LWP::Version, and inserted this line before it:
```
 $LWP::Version = $VERSION;
```
The differences (according to "diff"), compared with the original, were as follows:
```
 530c530,531
 < $main::ua->agent("Checkbot/$VERSION LWP/" . LWP::Version);
 ---
 > $LWP::Version = $VERSION;
 > $main::ua->agent("Checkbot/$VERSION LWP/" . $LWP::Version);
```
After that I was able to run "make install" successfully.
Crawl The TEST Instance: Checkbot is a robust and efficient little bot that uses the LWP package. When I dispatched it on its first run, it figured out that the server it had been instructed to crawl was not actually the Internet host it purported to be. It reported that it understood this and then set about crawling the site. After some initial testing I was able to set up a shell script to drive it. I included a list of URL branches to ignore in order to avoid potential "runaway" conditions if it strayed too close to any database rabbit holes. Note: There is also an option to include a pause between each fetch, which would be useful if you were crawling a PROD site. Checkbot writes a couple of HTML files as output. These can be handy for examining the results; however, I also included the --verbose option. This option directs checkbot to write a report to STDERR. I used this data as input for the next step, which would chase down files that contained the suspect URLs.
Fix The Broken Links: I constructed a perl script, called "fix_broken", to read data from the report produced by checkbot with the --verbose option, load it into a hash, and search for files on the server back-end that contained the offending URLs. If it found one it would replace the URL with a non-clickable , leaving the original anchor text in situ. To illustrate, let's consider an example ... Suppose your HTML code contained a reference to the famous "Fred Nurk" and his research into supercilious amphibians. In which case, your code might look like this:
```
You can read about <a href="http://nurk.com/supercilious_amphibian">Fred Nurk</a> and his work for more details.
```
After running the fix_broken conversion script, this would be changed to the code below:
```
You can read about Fred Nurk and his work for more details.
```
This cleaned up hundreds of broken links, but left a few stragglers. The checkbot run converts links when it looks them up. For example, Fred might have had a few domains, and perhaps one of them was called "nurkusa". However, it was shared as "nurkUSA". Checkbot would convert it back to "nurkusa" for lookup and for the report. There are other subtle differences in the way that URLs are stored and reported. By tweaking the fix_broken script I was able to deal with most of them. In order to clean up the last few by hand, I modified the perl script so that it could be called as a filter, parsing input from STDIN. I then called it from within vi. When operating in this mode it would read the paragraph it was presented with, replace the first URL it found with a non-clickable block, and leave the remainder of the paragraph untouched.

After this was complete, the deprecated internal links had been fixed or removed, and the broken external links had been converted into non-clickable blocks that preserved the anchor text, thus maintaining the integrity of the text.

### Cleaning Up HTML

This would be much more of a chore. As far as I could tell, I would need to fix:

W3C Compatibility: When I set the site up, I had made sure that it complied with W3C standards. I had included a W3C validator on most of the key pages, and I would include W3C validation as the final step after committing to PROD. It has to be done in PROD because TEST is not publicly accessible. However, since I had made an effort to keep everything W3C-compliant, if I saw an error after putting a page into PROD, it was usually due to a few misplaced or unclosed tags and was easy to fix. However, after I stopped paying much attention to the site, I became less scrupulous, as I occasionally added content in my spare time. Errors started to develop and some of them were in the back-end templates, which could lead to multiple errors throughout an entire range of pages.
Style Sheets: By 2014, HTML 5 had become the accepted web standard, and modern HTML has continued to evolve from there. HTML 4 still works, but it is recommended that deprecated elements be moved into style sheets. I have never really "groked" style sheets ... And I've got to say to any millennials reading this that I use the verb grok in its original sense, having learned it long before Elon Musk popularised the term again; I discovered grok when I first read Stranger In A Strange Land, in 1969; but, returning to the 21st century, even though I didn't grok them, I realised the advantages that style sheets offered, so I adopted them not too long after I set the site up. Generally I did this by copying style sheets and modifying them to work with my site. This was a rather tedious process of trial and error. Once they were working, I was reluctant to go back and change them. If I made a change it would often break something, so I decided it was better to "let sleeping dogs lie".

The task looked daunting. Fortunately, an AI chatbot came to my rescue. I collected samples using the "View Source" option in various browsers, included a copy of the style sheet, and uploaded them to Google's AI assistant. It came back in a few seconds with a detailed diagnosis of the errors, together with suggested fixes and improvements. The development cycle was so short that I became quite blasé about making changes that, in the past, might have taken me down a rabbit hole of gnarly CSS syntax brambles and cost me many hours of painstaking RTFM. I did have to break the task up into different categories of basic templates, and I decided to do it in stages rather than attempt all of the changes in one hit, but it was far easier with an AI assistant.

The final task was writing a script that would commit all of the changes. Many years ago, after I had completed the initial setup, I would change only a few pages at a time. I had created a bespoke script that would deploy the changes one at a time. I decided to keep the script because it still worked, but I needed something to feed it. I now had hundreds of individual items that had to be deployed. I didn't need a complex code management system that would take hours, or possibly days, to implement and manage, so I created a bash script to drive the deployment process.

Lastly, I started to create some content. This may turn out to be an exercise in futility because, in our brave new AI-generated universe, very few users visit text-based websites. However, I pressed on. AI once again proved very useful for editing text, picking up spelling and grammar errors, and fact-checking.

The website is still very much a work in progress. There are undoubtedly more rough edges to smooth out, but the heavy lifting is now complete. What would once have consumed months of painstaking investigation has been reduced to an iterative conversation with an AI assistant. Whether that ultimately proves to be a blessing or a curse for the craft of software development remains an open question, but for this project it has been an undeniably useful one.