LLM-assisted Abbreviation Mining for Legacy Systems

Markus Harrer

Introduction

Software archaeology

“Archaeologists try to find the clues left by people who lived before us, and they try to make sense of them.” (Source). This work is no different for software developers maintaining older systems, who venture as software archaeologists into old codebases like adventurers in a long-forgotten city. Many tasks are the same, such as deciphering the hieroglyphs (aka cryptic abbreviations) left behind by the original creators, unlocking valuable knowledge hidden beneath layers of history. And just as traditional archaeologists piece together ancient stories, software archaeologists help to understand the stories of legacy systems that have stood the test of time, and paving the way for a successful modernization journey.

Analysis goal

The goal of this archaeological analysis is to decipher the dense and often obscure language that developers used to describe their domain, processes, and business rules within a legacy codebase, gaining an initial understanding of what the code is about. This analysis uses Large Language Models (LLMs) to assist in uncovering the fundamental concepts buried within a legacy COBOL codebase, focusing specifically on mining abbreviations and other cryptic identifiers from filenames.

Why does this work? Because abbreviations used in filenames often act as shorthand for business terms, functional areas, framework specifics, or even system modules. They are typically used consistently throughout the codebase due to established (explicit or implicit) naming conventions. Recognizing, grouping, and extending these abbreviations to their original meanings helps create an initial understanding of the system, including its domain-specific terminology and the various functional areas it might cover.

The analysis in this blog post can serve as a first step towards a deeper comprehension of the codebase, helping to identify potential areas of interest, critical functionalities, and candidate modules for further examination or refactoring. It aims to reduce the time and effort required to make sense of legacy software and provides a foundation for upcoming modernization tasks.

Limitations

There are also limitations when using this approach. While LLMs might excel at pattern recognition and can suggest plausible expansions for abbreviations based on context, they still operate without true domain understanding and may hallucinate. This can lead to incorrect interpretations, especially in a specialized or highly customized business domain like COBOL mainframe systems, where abbreviations may have evolved based on internal company jargon or historical practices rather than standard terminology.

Moreover, relying only on abbreviation mining from file paths and names may provide a fragmented view of the codebase. There might be insufficient context, as different parts of the code may use similar or overlapping terms with entirely distinct meanings. This can result in ambiguous or misleading concept identification without careful cross-checking against the actual code logic and domain knowledge.

But I think this approach is better than having absolutely no clue what the codebase is about. Combining the automated identification and expansion of relevant abbreviations (including their possible meaning) with manual inspections using domain expertise opens up plenty opportunities to speculate about the system’s design and is absolutely no rocket science (as you’ll see in this blog post).

Today’s subject under inspection

I attempted to find and analyze a real legacy codebase, but obtaining a realistic scenario was challenging. This is where the “Mainframe CardDemo Application” (GitHub) from AWS comes into play: This project is a sample application designed to showcase modernization strategies for mainframe workloads using AWS services. It features a typical banking scenario with COBOL programs, JCL scripts, and related data files. The demo project should provide a realistic environment to demonstrate refactoring, replatforming, and migration of legacy code to cloud-based solutions. Although the number of files in this project is small, I believe it is a realistic starting point for exploring an unfamiliar legacy application written in a programming language I’m not familiar with.

Analysis

Step 1: Clone the repository

To begin, we clone the source code repository:

git clone https://github.com/aws-samples/aws-mainframe-modernization-carddemo.git

Then, we get a nice and clean file list we want to work with.

import glob

root_dir = "../aws-mainframe-modernization-carddemo/app/"
glob_list = glob.glob(f"{root_dir}**/*.*", recursive=True)
file_list = [f.replace("\\","/").replace(root_dir, "") for f in glob_list]
file_list[:5]

This give us a list with paths (only first five entries shown):

['bms/COACTUP.bms',
 'bms/COACTVW.bms',
 'bms/COADM01.bms',
 'bms/COBIL00.bms',
 'bms/COCRDLI.bms']

Step 2: Set up LLM assistant

I use Claude Sonnet 3.5, a large language model from Anthropic, to expand the abbreviations. For this type of application, I always set a low temperature to get consistent, reproducible results across different runs. I’ve also noticed that LLMs tend to get lazy when working on tasks I delegate because I don’t want to do them myself. Perhaps my own approach is rubbing off on the model. That’s why I make sure to remind the LLM not to be lazy in this case.

import anthropic
client = anthropic.Anthropic()

def ask(prompt):
    return client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=2000,
        temperature=0.0,
        system="""
        You're a software archaeologist who tries to make sense of the past.
        Respond in short and clear explanations.
        Don't be lazy!""",
        messages=[{"role": "user", "content": prompt}]
    ).content[0].text

Step 3: Define the base prompt

The base prompt was designed to guide the LLM in extracting and expanding abbreviations. I paid attention to providing clear, step-by-step instructions to help the LLM through abbreviation extraction and concept definition. It includes handling alternative meanings and uncertainties by suggesting confidence scores and providing alternative entries. I also requested simple regex patterns for practical file matching, useful for locating relevant files later on. I specified a structured output as a JSON schema for consistency and easier integration with subsequent analysis. To my knowledge, Sonnet doesn’t have a built-in option for this kind of structured output, so I also included an example to clarify expectations (and to avoid breaking the following code).

base_prompt = """

Below is a list of paths from a software program containing numerous abbreviations.

Your task is to build a list of abbreviations and a glossary entry of their corresponding concepts.
For this, determine the meaning of an abbreviation found in the paths and filenames.
For very similar abbreviations, make a separate entry. Don't put more than one abbreviations in one entry.

Then, provide a definition of each concept and provide the information if it is a business concept or a technical concept.
With that, also estimate a confidence score between 0 and 1 indicating how certain you are about the term and its definition.

Additionally, find a simple regular expression that identifies all files related to the abbreviation (and their alternative spellings).

If there are not more abbreviatios to discover, deliver back an empty JSON list.

Output only and directly as a JSON array (not with a key) using strict the following schema:
- 'abbreviation': The abbreviation.
- 'meaning': The meaning of the abbreviation.
- 'type': business, technical
- 'definition': An explanation of the concept.
- 'regex': A regular expression to locate related files.
- 'confidence': A value between 0 and 1 indicating certainty.
- 'alternative': An alternative meaning, or "-" if none exists.

Example output:
[
    {
      "abbreviation": "ABC",
      "meaning": "Air Bullet Container",
      "type": "technical"
      "definition": "A storage unit used for air bullets in testing scenarios.",
      "regex": ".*ABC.*",
      "confidence": 0.8,
      "alternative": "-"
    }
]

Now, expand the following abbreviations to their full meanings:
"""

Step 4: Assemble the initial prompt

Next, I combine the base prompt with the file list from above. I intentionally shuffle the filenames to avoid any biases and (hopefully) keep the LLM engaged with a varied input. The shuffled list is then appended to the base prompt for analysis. Concatenating the shuffled list with the base prompt creates a single string where each file path appears on a new line, making it easier for the model to process.

import random

random.shuffle(file_list)
prompt = base_prompt + "\n".join(file_list)
print(prompt[:100] + "...")

The prompt looks like this:

Below is a list of paths from a software program containing numerous abbreviations.
    
Your task is ...

Step 5: Ask Claude via backfeeding

The easy part here is that I simply send the prompt to the LLM and hope that we get the results as a JSON data structure (fingers always crossed). In more detail, this process leverages an iterative feedback loop to refine the extraction of abbreviations. After each pass, it updates the prompt by including abbreviations already identified, guiding the AI to focus on new terms and avoid redundancy. The analysis continues as long as the extracted meanings maintain a high confidence level, stopping when confidence drops below a set threshold.

import json

min_confidence = 1.0

json_results = []

backfeed = ""

while min_confidence > 0.7:

    evolving_prompt = prompt + "\n\n" + backfeed

    res = ask(evolving_prompt)
    json_result = json.loads(res)

    if len(json_result) == 0:
        break

    min_confidence = min([i['confidence'] for i in json_result])
    backfeed = "\n\nI already discovered these abbreviations that I don't need anymore:\n" + "\n".join([i['abbreviation'] for i in json_results])
    json_results.extend(json_result)
    
print(str(json_results)[:100])

That’s what the result looks like:

[{'abbreviation': 'CBL', 'meaning': 'COBOL', 'type': 'technical', 'definition': 'Common Business-Ori

Step 6: Prepare for first analyses

For easier inspection of the result (and later analysis steps), I like to load the expanded abbreviations and their meanings in a pandas DataFrame.

import pandas as pd
abbreviations = pd.DataFrame.from_dict(json_results)\
    .sort_values(by='abbreviation')\
.drop_duplicates(subset=["abbreviation", "meaning"])
abbreviations.head()

Here are the first five entries of the DataFrame:

Step 7: Connect files on found patterns

Next, I search for all files that correspond to the pattern in the regex column. This allows us to map the actual files to specific abbreviations or concepts. I iterate through the list of known abbreviations, using their regular expressions to find matching files in the project. For each abbreviation, I refine the regex, apply it to filter the file list, and update the list of unmatched files. The matched files and their proportion of the total file count are stored for each abbreviation, and the results are added to the DataFrame for analysis. However, this approach isn’t entirely clean, as files can belong to multiple concepts or match more than one abbreviation, leading to potential overlaps. But I think this is good enough for now.

files = pd.Series(file_list)

abbreviation_files = []
proportions = []

length_all_files = len(file_list)

for i, entry in abbreviations.iterrows():
    # remove possible capture groups and start markers
    regex = entry['regex'].replace("(", "(?:").replace("^", ".*")
    files_found = files[files.str.contains(regex)].sort_values().to_list()
    
    proportions.append(len(files_found)/length_all_files)
    abbreviation_files.append(files_found)

abbreviations['prop'] = proportions
abbreviations['paths'] = abbreviation_files
abbreviations.head()

Here are the first five entries of the DataFrame. Click on the info box below for the whole table.

Step 8: Coverage of abbreviation information

This step calculates the proportion of files in the project that contain identifiable abbreviation information. The resulting value indicates the percentage of files for which we were able to extract meaningful details based on the identified abbreviations.

files_with_info_about_abbreviations = len(abbreviations.explode('paths')['paths'].drop_duplicates())
coverage = files_with_info_about_abbreviations / length_all_files
coverage_text = f"{coverage*100:.2f}% of the files contain one or more abbreviations of concepts we know about"
print(coverage_text)

This gives us this result:

99.31% of the files contain one or more abbreviations of concepts we know about

Assessment

Step 9: Generate assessment report

At this point, a software archaeologist can already get to know the system and reason about it. But this could be a tedious job for larger applications. So why not generate a summary of the results also via the LLM? That’s exactly what I’m doing here:

from IPython.display import display, Markdown

assessment_prompt = f"""
Here is a table with the information about a COBOL application.
Find the key insights for a software archeologist and summarize the findings in this assessment:
    
{abbreviations.to_markdown()}

Also: {coverage_text}
"""

result = ask(assessment_prompt)
display(Markdown(result))

This generates an assessment report of the results:

Here’s a clear archaeological assessment of this COBOL application:

Key Findings:

System Type

• This is a Credit Card Management System running on IBM mainframe
• Uses CICS for transaction processing
• Handles customer accounts, cards, and financial transactions

Technical Architecture

• Core components: COBOL programs (.cbl), CICS screens (.bms), copybooks (.cpy)
• Data stored in EBCDIC format physical sequential files (.PS)
• Batch processing through JCL jobs
• Heavy use of copybooks (31% of codebase) suggesting modular design

Main Business Domains

• Account management (ACCT*)
• Card operations (CRD*)
• Customer data (CUST*)
• Transaction processing (TRAN*)
• User security (USR*)
• Billing/Statements (BIL, STM)

Notable Patterns

• Consistent naming conventions (CO prefix for programs)
• Clear separation between business and technical components
• Strong batch processing component (20% JCL)
• Comprehensive user interface (multiple BMS screens)

System Maturity

• Well-structured with clear naming conventions
• High confidence in identified abbreviations (99.31% coverage)
• Complete mainframe ecosystem (online + batch)
• Comprehensive security and user management

This appears to be a mature, production-grade mainframe application following standard IBM mainframe architectural >patterns of its era.

(Output slightly formatted for the blog post version for better readability)

OK, there might be some mistakes regarding the interpretation. But I think it is overall a great overview of the software system at hand.

Summary

In this analysis, I applied a LLM-assisted approach to decipher abbreviations embedded in a legacy codebase, focusing on enhancing our understanding of key business and technical concepts. By leveraging a well-structured prompt and an iterative feedback loop, I extracted abbreviations, expanded their meanings, and linked them to relevant files using regular expressions. The high coverage of identifiable abbreviations indicates a comprehensive grasp of the code’s structure and purpose. This iterative method allows us to refine the analysis based on previously discovered information, avoiding redundancy and increasing the accuracy of the interpretations. The generated insights gained can provide a solid foundation for upcoming modernization efforts and help understanding the critical business logic of a legacy application.

What do you think? Can this be useful for your legacy application as well? Let me know!

You can find an earlier version of this article as a Jupyter Notebook on GitHub