Introduction
The Historical Index Conversion Tracker (HICT) was developed to help track the
conversion of scanned Historical Index pages into readable and editable
electronic copies. During early work on this project Kapala IT utilized several
time-consuming and error prone processes. As the project continued we were able
to identify the causes of these issues being heavily related to the
distributing and tracking of work among multiple employees. The HICT helps to
alleviate these problems by presenting an easy to use and secure application
that tracks every useful piece of information related to the project.
System Requirements
- Python 2.7 - Any version 2.7.x is acceptable. This tool has not been tested with Python 3 and therefore cannot be guaranteed to work with it. (must be added to PATH)
- Flask - Any version 0.12.x is acceptable.
- Node.js - Necessary to run the startup commands from the console. (must be added to PATH)
- Electron - Allows us to build a desktop application using web languages.
Useful Documents
Overview
The HICT is a desktop application was built using Electron
for the front-end and Python for the back-end.
The front-end consists of a main process and five different
renderer processes. The main process runs behind the scenes to keep the
application functioning, while the renderer processes display a user interface
and handle interactions between the user and the back-end. Before updating any
files make sure to understand what each file type is used for:
- .html – Static contents of the web page.
- .js – Handles events triggered by the user
(button clicks, key presses, etc.), updates the html contents, communicates
with the back-end,
- .css – Styling information, how the page looks, what
fonts, font sizes, colors, margins, etc. are used.
There are other files within the project that are not one of
the above types, but it is unlikely that you will ever need to update these
files. You should have a decent grasp on the basics of each language
and the main source code before attempting to make any changes.
The back-end is coded purely in python using the sqlite3
module for database operations.
- .py – Receives requests from the front-end via
flask module, interacts with the database via sqlite3 module, returns requested
data or errors to the front-end.
- .sqlite – Database file, operated on using
python files, this file is not used directly.
You should never have to edit the SQLite file directly, and
to do so would require a third-party program that can read and edit SQLite
database files. If you need to make changes to the database, make sure to
carefully examine the structure of the database and python code before doing
so. Once you have, research the sqlite3 library and the SQLite language in
general to understand how to use queries, transactions, etc.
Front-End
In this section, we’ll go over in detail the structure of
each element of the front-end. The main process is contained within one file,
while each of the renderer processes is split between two files.
Main Process
Files: main.js
Responsibilities: Handles the life-cycle of the application.
How to start, when to clean up. Provides methods to the renderer processes.
Login Renderer Process
Files: login.html, login.js
Responsibilities: Displays the login page and sends log in requests.
Corrections Renderer Process
Files: index.html, index.js
Responsibilities: Displays the corrections page and sends check-in and check-out requests.
Quality Control Renderer Process
Files: quality_control.html,
quality_control.js
Responsibilities: Displays the Quality Control page, sends qc approval requests, and allows for the package files to be viewed and
updated.
Admin Renderer Process
Files: admin.html, admin.js
Responsibilities: Displays the admin page and sends requests to import packages and create new notifications for the corrections page.
Back-End
The back end consists of two files, a python file that uses the flask module to handle requests from the front-end, and an SQLite3 database file that the python file accesses to read or write data.
Python
Files: database_handler.py
Responsibilities: Handles all requests to read and write data from the front end. Handles all interactions with the database.
Required Non-Standard Python Modules: flask
For an in depth look at which python function handles which request and how, take a look at the source code and its documentation. In this section we'll quickly go over the general structure in this file.
Each python function in this file that handles a request from the front end has a decorator above the function declaration. This decorator will look something like this:
@app.route("/some_name/<some_variable>/<path:some_path>"
def some_name(some_variable, some_path):
Here is a quick breakdown of this decorator
- @app.route - Flask routes the requests from the front end to a specific python function using this decorator.
- some_name - This is the first part of the request, it acts as an identifier for this function, it can be anything you want but generally it should be the name of the python function or something similar.
- some_variable/path:some_path - After you've added the name to the request you may want to also pass some data, using this syntax you can easily pass this data using python parameters. For example, if you have <some_variable> in your decorator string you can then have some_variable as a parameter with the value that was passed in during the request. These parameters are ordered based on the ordering in the request. That means in our above example we first get some_variable and then some_path so our python function would order the parameters respectively.
Knowing this, finding which function handles a specific request is quite simple. All you need to do is search for the request name (some_name in our example above).
The general structure of each python function is the same. They follow the following format:
- receive request and any parameters.
- interface with files or the database.
- return response in JSON string format (usually).
SQLite3
Files: hict.sqlite
Responsibilities: Stores all data related to the tracking of the conversion process.
General formatting rules:
- Dates use the 'yyyy-mm-dd hh:mm:ss' format. ex. '2017-01-27 12:05:38'
- If multiple flags apply to a single page, they will be seperated by a vertical bar character. ex. 'Difficult to read|Wyoming verify data'
Below you will find detailed explanations of the tables included in this database.
corrections
Holds information related to the correction status of each package.
corrections Table Definition
Name | SQLite3 Type | Description |
PackageId | TEXT | The unique identifier attached to each package. |
Username | TEXT | The username of the user who checked out the package. |
CheckOutDate | TEXT | The date on which the user checked out the package. |
CheckInDate | TEXT | The date on which the user checked in the package. |
credentials
Holds user credentials and other important information.
credentials Table Definition
Name | SQLite3 Type | Description |
username | TEXT | The user's username. |
firstname | TEXT | The user's first name. |
lastname | TEXT | The user's last name. |
email | TEXT | The user's email address. |
userclass | INTEGER | The user's user class. Determines permissions. |
password | TEXT | The user's password. Stored as a hashed string. |
Restricted fields
- userclass: follows the guidelines below. userclass Values
Value | User Level | Page Rights |
0 | Corrections | Corrections |
1 | QC | Corrections and QC |
2 | Admin | Corrections, QC, and Admin |
- password: stored as a hashed value. See the login function within the python source code for more details.
error_pages
Holds page data for all pages with errors that stop them from going any further in the conversion process.
error_pages Table Definition
Name | SQLite3 Type | Description |
PageId | TEXT | The unique identifier attached to each page. |
PackageId | TEXT | The unique identifier attached to each package. |
Flags | TEXT | The error flags attached to the page. |
Restricted fields
imports
Holds information related to the import of each package.
imports Table Definition
Name | SQLite3 Type | Description |
PackageId | TEXT | The unique identifier attached to each package. |
Username | TEXT | The username of the admin user that uploaded the package. |
Date | TEXT | The date on which the admin user uploaded the package. |
notifications
Holds information on all notifications.
notifications Table Definition
Name | SQLite3 Type | Description |
NotificationId | INTEGER | Auto-generated integer used to uniquely identify a notification. |
Message | TEXT | The message displayed in the notification. |
Username | TEXT | The username of the admin user who added the notification. |
Date | TEXT | The date on which the admin user added the notification. |
packages
Holds general tracking information on each package not related to a specific stage.
packages Table Definition
Name | SQLite3 Type | Description |
PackageId | TEXT | The unique identifier attached to each package. |
PLSSID | TEXT | The PLSSID attached to the package. |
NumPages | INTEGER | The number of pages in the package. |
pages
Holds page data for all pages besides those that have been removed from the conversion process due to severe errors.
pages Table Definition
Name | SQLite3 Type | Description |
PageId | TEXT | The unique identifier attached to each page. |
PackageId | TEXT | The unique identifier attached to each package. |
Flags | TEXT | The error flags, if any, attached to the page |
Restricted fields
qc
Holds information related to the QC status of each package.
qc Table Definition
Name | SQLite3 Type | Description |
PackageId | TEXT | The unique identifier attached to each package. |
Username | TEXT | The username of the QC or admin user who QC'd and approved the package. |
Date | TEXT | The date on which the QC or admin user approved the package. |
sqlite_sequence
Auto-generated and updated table that keeps track of the ASSIGNEDIDS auto increment variable.
sqlite_sequence Table Definition
Name | SQLite3 Type | Description |
name | TEXT | The name of the table whose sequence is being tracked. |
seq | INTEGER | The current value of the sequence. |
Restricted fields
- Do not edit this table. It is automatically handled by SQLite.