NLP development and evaluation requires tools to compare annotations, analyze differences, and compute accuracy metrics.
None of the few existing tools allows consuming the output of any NLP system, comparing it with reference annotations in any format (both with configuration), setting comparison rules, counting matches and computing accuracy metrics, visually displaying matching and erroneous annotations, changing the displayed information for analysis on the fly, exporting evaluation results, etc.
ETUDE (Evaluation Tool for Unstructured Data and Extractions) is a collaborative development by the MUSC TBIC (Paul Heider, for the engine) and by Clinacuity, Inc.
It offers the following functionalities:
- Import of annotations (system output and reference standard) in multiple configurable formats
- Drag-and-drop creation of evaluation configuration files (i.e., specification of annotation categories and attributes to match and compare)
- Customizable annotation comparison settings (base includes exact, partial, and fully-contained matches)
- Count and comparison of annotations, computing descriptive statistics, a confusion matrix of matches and mismatches, and computing accuracy metrics (recall, precision, F-measure)
- Show side-by-side reference and system annotations in the original document context with easy exploration of matches and errors
- Export of evaluation results in various configurable formats
ETUDE is available with an open source license and available for download on GitHub at https://github.com/MUSC-TBIC/etude-engine and https://github.com/Clinacuity/etude-viewer.
Press release available at http://www.scra.org/media-center/new-sc-launch-client-companies-jan
Summerville, S.C. – SCRA announced the recent acceptance of five Client Companies into its entrepreneurial program, SC Launch. Client Companies receive mentoring and support services and may also be eligible for grants, matching funds and investments.
Clinacuity’s solutions extract critical information from patient electronic healthcare records. The data extracted provides clarity and efficiency to improve patient care as well as provides de-identified information to researchers that will enhance treatment outcomes. The company is developing two product prototypes with funding from the National Cancer Institute and the National Institute of General Medical Sciences with clinical impact evaluations by Hollings Cancer Center at the Medical University of South Carolina and the University of Utah.
Clinacuity has been selected to participate in the biennial NIH Commercialization Accelerator Program (CAP) for 2017-2018, an excellent opportunity for our company!
The CAP is designed to help NIH technology transfer funding awardees transition their SBIR/STTR developed products into the marketplace through a contract with Larta Inc.
It will provide companies with individualized assistance toward accomplishing their commercialization goals.
Clinacuity has been selected as one of the MedTech Innovator Top 100 companies.
In March 2017, Clinacuity opened new offices at The Schoolhouse (an old elementary school transformed in office spaces) in Charleston, SC, with a new team of developers. A great place!