Track % of code generated in yaml file - automatically measuring WCA adoption

NOTE: This relates to another of my ideas, using (G) as an identifier for generated content: https://ideas.ibm.com/ideas/WCAST-I-297

Explanation:

With a short (G) identifier for generated code, there's no indication which model was used or whether it's approved for use with that language.

But, that can be documented elsewhere. Adding a doc to the root of the code repository, perhaps GENAI.md, or Genai-manifest.yaml, and managing it with a scripted automation. I think that should be sufficient for compliance.

With a doc like that, there is lots of potential for automation or CI/CD workflows:

• Get a % generated code indicator on GitHub

• trigger report creation

• Alert code reviewers to take an extra look at generated parts

• etc

The automation could be bundled with the VScode WCA extension. Or just a standalone CLI app that can be open-sourced on its own to encourage widespread adoption.

Effort?

Not super high. Apart from a few rare cases, '(G)' is somewhat unique, and be found fast by regex on all files. That gives a rough estimate. Going more fine-grained than that would need implementing scoping, and some other rules i.e. a function's body being considered generated if its doc string has '(G)' in it. Since only 3(?) programming languages are approved for WCA generation currently, there is not too much work to do to adapt it. Which programming language to assume can be inferred from filename endings.

Example:

> wca_track .
Tracking generated code in /path/to/repo/..., matching tag '(G)'...
---------------------------------------------------------------------

Scanning files for AI-generated code...
Approved languages: Python, Java, TypeScript
Files with AI-generated code found:

- src/utils/data_processing.py  | Line 15  | (G) ibm-granite/granite-3b-code-base-2k
- src/modules/recommendation.py | Line 30  | (G) meta/llama3:vX.Y
- tests/test_model.py           | Line 10  | (G) ibm-granite/granite-8b-code-instruct-4k
- src/main.py                   | Line 50  | (G) watsonx/code-assistant:v1.2

Generating manifest file...

Manifest generated at: /path/to/repo/GenAiManifest.yaml

Report successfully created. You can generate a detailed report by running:

wca_track --report /path/to/repo/Genai-manifest.yaml

--------------------------------------------------------

Can be taken further:

• git pre-commit hook to automatically add (G) to the message if any line in commit + contains '(G)'

• etc...