Automated Book Builder Sample Feature Brief

The Context & Challenge

With a growing library and output formats, manually generating every book and every output format for testing becomes time-consuming. We also have more reliable pipelines where the majority of output formats do not have a lot of regressions, therefore a lot of time is wasted checking on output formats that have little to no changes. Regressions are gradually becoming a needle in a haystack situation.

The Approach

First, our Engineering Manager did a brief 1 week spike to see if the timing was good for this type of work for the team and to see what technical options we could have. From there I was able to determine a plan and write up a feature brief with the help of our Engineering Manager. I then consulted users of this tool for their feedback.

The Feature Brief

Description

This is for a new CLI tool to pull the entire library and run automated system testing to check for formatting differences and generation failures.

What is the problem?

With a growing library and output formats, manually running every book and every output format for testing becomes time-consuming. We also have more reliable pipelines where the majority of output formats do not have a lot of regressions, therefore a lot of time is wasted checking on PDFs that have no changes.

Who has the problem?

CE developers, CE QA, and Content Managers

How do we know this is a real problem worth solving? (And why now?)

With the release of the new MacBooks, we have an opportunity for everyone to have the hardware upgrade to run a tool like this. With languages and various tools, we use release new versions, dependency upgrades are a case where the potential for unusual side effects makes it worthwhile to check the whole library. An example is that we are still on Ruby 2 and upgrading to Ruby 3 will provide more opportunities for us to take advantage of new tools. Not having a good way to do pan-library checks is a potential blocker for many ways the pipeline could grow & change.

What are we currently doing about these problems?

We are using our bare hands to generate every PDF and using our eyeballs to eye changes. We can at least diff HTML but have to generate everything by hand.

How do we know if we’ve solved this problem?

Test fests will be reduced in scope and we wouldn’t be catching issues super late. Changes with broad or unknown scope are possible to test.

What are the trade-offs and risks? Why are these tolerable?

Maintenance might be an issue if we add more things into the pipeline. It is one more tool to update.

What might this look like in the product?

User Stories

As content QA, I’d like to be able to generate the entire library automatically into whatever format I want (so long as it is supported).

As content QA, I’d like to be able to automatically compare two versions of pipelines (cookbook, styles, web hosting pipelines, or Enki) to see if there are any changes.

As content QA, I’d like to get a report on which book(s) failed and the location of failure so that I can troubleshoot and know what errors to look at more closely.

V1

  • Exploration/spike of what could be possible. First, see how we should pull in our library of books and auto-generate various output formats. Then designing CLI outputs.

Acceptance Criteria

  • generate XHTML (web view) and PDFs for books on ABL (approved books list) and in production

  • “in production” is defined as the private repositories where vendors or content managers are working on a yet-to-be-released book

  • a report that tells you what generated or failed

  • detailed logs and output formats stored in your machine

  • locally done

  • portable! meaning code can be picked up from one computer and put onto another computer

  • needs to be able to run on Linux, M1 MacBooks, regular old MacBooks, and Windows

  • should allow users to continue editing/running enki without affecting the BABBY run

  • output has a summary of books that failed in addition to succeed and failure counts

  • ideally should generate under 10 minutes per book**

  • should be able to run in the background without disrupting other tasks on your machine**

  • **optimization is a nice to have and will be prioritized in v4; something to think about for the future while architecting tool for v1

V2

Exploration to see if acceptance criteria are possible to achieve and identify diffing tools that our teams can potentially use. Need to also design logs and error messages that are informative and easy to understand.

Acceptance Criteria

  • customization options for testing: need to be able to point the following pipelines to two different code versions of submodules

  • building the library twice: need to be able to generate and store web view/PDF/EPUB outputs for both versions

  • need a way to map old and new versions

  • generation failures should still be noted (from V1)

V3

Acceptance Criteria

  • take outputs from above (from the mapping document)

  • use diffing tools for various output formats

  • generate a log - priority is to log failures

V4

Acceptance Criteria

  • optimization items listed in v1

Other things to consider

  • Should this be an external service? (Currently it lives in our generation pipeline.)

  • Would we want to move away from a CLI tool in the future?

Previous

Mini-Demo: Underline Accessibility